{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 简介\n",
    "\n",
    "本文我们介绍一种能够较为稳定提升回归精度的方法,方法的思路非常简单,主要就是捕捉边缘值的预测,我们知道平方损失（MSE）的函数对于较大值的惩罚较大,而优化MAE的指标则比MSE更加注重较小值的优化,所以本文我们就以《消费者人群画像—信用智能评分》比赛为例，给出该方法的线下&线上的实践，来一起见证一下该想法是否正确。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 工具包导入&数据读取\n",
    "## 工具包导入"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/zjpy36/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "## 数据工具包\n",
    "import numpy as np\n",
    "np.random.seed(42)\n",
    "import pandas as pd\n",
    "from tqdm import tqdm,tqdm_notebook \n",
    "\n",
    "## 字符串处理工具包\n",
    "import string\n",
    "import re\n",
    "import gensim\n",
    "from collections import Counter\n",
    "import pickle\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.decomposition import TruncatedSVD \n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.model_selection import KFold\n",
    "from keras.preprocessing import text, sequence \n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import xgboost as xgb\n",
    "import lightgbm as lgb\n",
    "from functools import partial\n",
    "\n",
    "import os \n",
    "import gc\n",
    "import joblib\n",
    "from scipy import stats \n",
    "from scipy.sparse import vstack  \n",
    "import time\n",
    "import datetime\n",
    "import multiprocessing as mp\n",
    "import seaborn as sns \n",
    "tqdm.pandas() \n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据读取"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.read_csv('train_dataset.csv')\n",
    "test = pd.read_csv('test_dataset.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>用户编码</th>\n",
       "      <th>用户实名制是否通过核实</th>\n",
       "      <th>用户年龄</th>\n",
       "      <th>是否大学生客户</th>\n",
       "      <th>是否黑名单客户</th>\n",
       "      <th>是否4G不健康客户</th>\n",
       "      <th>用户网龄（月）</th>\n",
       "      <th>用户最近一次缴费距今时长（月）</th>\n",
       "      <th>缴费用户最近一次缴费金额（元）</th>\n",
       "      <th>用户近6个月平均消费值（元）</th>\n",
       "      <th>...</th>\n",
       "      <th>当月是否景点游览</th>\n",
       "      <th>当月是否体育场馆消费</th>\n",
       "      <th>当月网购类应用使用次数</th>\n",
       "      <th>当月物流快递类应用使用次数</th>\n",
       "      <th>当月金融理财类应用使用总次数</th>\n",
       "      <th>当月视频播放类应用使用次数</th>\n",
       "      <th>当月飞机类应用使用次数</th>\n",
       "      <th>当月火车类应用使用次数</th>\n",
       "      <th>当月旅游资讯类应用使用次数</th>\n",
       "      <th>信用分</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a4651f98c82948b186bdcdc8108381b4</td>\n",
       "      <td>1</td>\n",
       "      <td>44</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>186</td>\n",
       "      <td>1</td>\n",
       "      <td>99.80</td>\n",
       "      <td>163.86</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>713</td>\n",
       "      <td>0</td>\n",
       "      <td>2740</td>\n",
       "      <td>7145</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>aeb10247db4e4d67b2550bbc42ff9827</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>29.94</td>\n",
       "      <td>153.28</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>414</td>\n",
       "      <td>0</td>\n",
       "      <td>2731</td>\n",
       "      <td>44862</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>530</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5af23a1e0e77410abb25e9a7eee510aa</td>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>145</td>\n",
       "      <td>1</td>\n",
       "      <td>49.90</td>\n",
       "      <td>109.64</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3391</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4804</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>643</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>43c64379d3c24a15b8478851b22049e4</td>\n",
       "      <td>1</td>\n",
       "      <td>55</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>234</td>\n",
       "      <td>1</td>\n",
       "      <td>99.80</td>\n",
       "      <td>92.97</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>500</td>\n",
       "      <td>0</td>\n",
       "      <td>1931</td>\n",
       "      <td>3141</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f1687f3b8a6f4910bd0b13eb634056e2</td>\n",
       "      <td>1</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>76</td>\n",
       "      <td>1</td>\n",
       "      <td>49.90</td>\n",
       "      <td>95.47</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>522</td>\n",
       "      <td>0</td>\n",
       "      <td>64</td>\n",
       "      <td>59</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>648</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                               用户编码  用户实名制是否通过核实  用户年龄  是否大学生客户  是否黑名单客户  \\\n",
       "0  a4651f98c82948b186bdcdc8108381b4            1    44        0        0   \n",
       "1  aeb10247db4e4d67b2550bbc42ff9827            1    18        0        0   \n",
       "2  5af23a1e0e77410abb25e9a7eee510aa            1    47        0        0   \n",
       "3  43c64379d3c24a15b8478851b22049e4            1    55        0        0   \n",
       "4  f1687f3b8a6f4910bd0b13eb634056e2            1    40        0        0   \n",
       "\n",
       "   是否4G不健康客户  用户网龄（月）  用户最近一次缴费距今时长（月）  缴费用户最近一次缴费金额（元）  用户近6个月平均消费值（元） ...   \\\n",
       "0          0      186                1            99.80          163.86 ...    \n",
       "1          1        5                1            29.94          153.28 ...    \n",
       "2          0      145                1            49.90          109.64 ...    \n",
       "3          0      234                1            99.80           92.97 ...    \n",
       "4          0       76                1            49.90           95.47 ...    \n",
       "\n",
       "   当月是否景点游览  当月是否体育场馆消费  当月网购类应用使用次数  当月物流快递类应用使用次数  当月金融理财类应用使用总次数  \\\n",
       "0         1           1          713              0            2740   \n",
       "1         0           0          414              0            2731   \n",
       "2         0           0         3391              0               0   \n",
       "3         1           1          500              0            1931   \n",
       "4         1           0          522              0              64   \n",
       "\n",
       "   当月视频播放类应用使用次数  当月飞机类应用使用次数  当月火车类应用使用次数  当月旅游资讯类应用使用次数  信用分  \n",
       "0           7145            0            0             30  664  \n",
       "1          44862            0            0              0  530  \n",
       "2           4804            0            0              1  643  \n",
       "3           3141            0            0              5  649  \n",
       "4             59            0            0              0  648  \n",
       "\n",
       "[5 rows x 30 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 简单特征工程\n",
    "\n",
    "下面的特征很简单，我也没对比做不做特征好不好,就是随意写了一些简单的一眼能看出来的特征,具体有没有用，大家自己验证......\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _simple_features(df_):\n",
    "    df = df_.copy() \n",
    "    df['次数'] = df['当月网购类应用使用次数'] +  df['当月物流快递类应用使用次数'] +  df['当月金融理财类应用使用总次数'] + df['当月视频播放类应用使用次数']\\\n",
    "                 + df['当月飞机类应用使用次数'] + df['当月火车类应用使用次数'] + df['当月旅游资讯类应用使用次数']  + 1\n",
    "        \n",
    "    for col in ['当月金融理财类应用使用总次数','当月旅游资讯类应用使用次数']: # 这两个比较积极向上一点\n",
    "        df[col + '百分比'] = df[col].values / df['次数'].values \n",
    "    \n",
    "    \n",
    "    df['当月通话人均话费'] = df['用户账单当月总费用（元）'].values / (df['当月通话交往圈人数'].values + 1)\n",
    "    df['上个月费用'] = df['用户当月账户余额（元）'].values + df['用户账单当月总费用（元）'].values\n",
    "     \n",
    "    df['用户上网年龄'] = df['用户年龄'] - df['用户网龄（月）']\n",
    "    df['用户上网年龄百分比'] = df['用户网龄（月）'] / (df['用户年龄'] + 1)\n",
    "     \n",
    "    df['近似总消费'] = df['用户近6个月平均消费值（元）'] / 6 * df['用户网龄（月）']\n",
    "    return df\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_fea = _simple_features(train)\n",
    "test_fea  = _simple_features(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "fea_cols = [col for col in train_fea.columns if train_fea[col].dtypes!='object' and train_fea[col].dtypes != '<M8[ns]' and col!='用户编码' and\\\n",
    "            col!='信用分']   \n",
    "\n",
    "len(fea_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 线下训练&验证\n",
    "## 线下验证函数\n",
    "\n",
    "- 我们认为MSE对于两端极值的预测较为准确,而MAE则对于中间的预测更为准确,所以我们对函数预测的极值附近进行简单的加权修正。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_absolute_error\n",
    "from sklearn.model_selection import StratifiedKFold,KFold\n",
    "from sklearn.metrics import mean_squared_error\n",
    "def _get_values_lgbregresser_models(df_fea, df_label,  feature_names):\n",
    "    kf = KFold(n_splits=5,shuffle=False)#,random_state=1)\n",
    "     \n",
    "    models  = [] \n",
    "    models_1 = []\n",
    "    models_2 = []\n",
    "    \n",
    "    importances = pd.DataFrame() \n",
    "    \n",
    "    lgb_params = {'num_leaves': 31,\n",
    "         'min_data_in_leaf': 32, \n",
    "#          'objective':'mae',\n",
    "         'max_depth': -1,\n",
    "         'learning_rate': 0.005,\n",
    "         \"min_child_samples\": 20,\n",
    "         \"boosting\": \"gbdt\",\n",
    "         \"feature_fraction\": 0.9,\n",
    "         \"bagging_freq\": 1,\n",
    "         \"bagging_fraction\": 0.9 ,\n",
    "         'n_estimators': 10000,\n",
    "         \"bagging_seed\": 11,\n",
    "         \"metric\": 'rmse',\n",
    "         \"lambda_l1\": 0.1,\n",
    "         \"nthread\": 50,\n",
    "         \"verbosity\": -1}\n",
    "\n",
    "\n",
    "    lgb_params1 = {'num_leaves': 31,\n",
    "         'min_data_in_leaf': 32, \n",
    "         'objective':'mae',\n",
    "         'max_depth': -1,\n",
    "         'learning_rate': 0.005,\n",
    "         \"min_child_samples\": 20,\n",
    "         \"boosting\": \"gbdt\",\n",
    "         \"feature_fraction\": 0.9,\n",
    "         \"bagging_freq\": 1,\n",
    "         \"bagging_fraction\": 0.9 ,\n",
    "         'n_estimators': 10000,\n",
    "         \"bagging_seed\": 11,\n",
    "         \"lambda_l1\": 0.1,\n",
    "         \"nthread\": 50,\n",
    "         \"verbosity\": -1}\n",
    "    \n",
    "    min_val = np.min(df_label)\n",
    "    print(min_val)\n",
    "    for fold_, (trn_, val_) in enumerate(kf.split(df_fea)): \n",
    "        trn_x, trn_y= df_fea[trn_,:], df_label[trn_]#, df_label1[trn_] \n",
    "        val_x, val_y = df_fea[val_,:], df_label[val_]#, df_label1[val_] \n",
    "        tmp = pd.DataFrame()\n",
    "         \n",
    "        \n",
    "        model = lgb.LGBMRegressor(**lgb_params1)\n",
    "        model.fit(trn_x, trn_y, eval_set=[(trn_x, trn_y), (val_x, val_y)], eval_metric ='mae',verbose=50,early_stopping_rounds=250)     \n",
    "        tmp['target'] = val_y\n",
    "        tmp['pred1'] = model.predict(val_x)\n",
    "        models.append(model)\n",
    "        \n",
    "        model1 = lgb.LGBMRegressor(**lgb_params)\n",
    "        model1.fit(trn_x, trn_y, eval_set=[(trn_x, trn_y), (val_x, val_y)], eval_metric ='mae',verbose=50,early_stopping_rounds=250)     \n",
    "        tmp['pred2'] = model1.predict(val_x)\n",
    "        models_1.append(model1)\n",
    "  \n",
    "        tmp = tmp.sort_values('pred1')\n",
    "        tmp['ranks'] = list(range(tmp.shape[0]))\n",
    "        tmp['preds'] = tmp['pred1'].values\n",
    "        tmp.loc[tmp.ranks<2000,'preds']  = tmp.loc[tmp.ranks< 2000,'pred2'].values *0.4 + tmp.loc[tmp.ranks< 2000,'pred1'].values * 0.6\n",
    "        tmp.loc[tmp.ranks>8000,'preds']  = tmp.loc[tmp.ranks> 8000,'pred2'].values *0.4 + tmp.loc[tmp.ranks> 8000,'pred1'].values * 0.6\n",
    "         \n",
    "        print('*' * 100)\n",
    "        print('MAE Model',     1 / (1 + (mean_absolute_error(y_true= tmp['target'] , y_pred= tmp['pred1'] ))))\n",
    "        print('MSE Model',     1 / (1 + (mean_absolute_error(y_true= tmp['target'] , y_pred= tmp['pred2'] ))))\n",
    "        print('Merge Model12', 1 / (1 + (mean_absolute_error(y_true= tmp['target'] , y_pred= tmp['preds'] )))) \n",
    "        \n",
    "        imp_df = pd.DataFrame()\n",
    "        imp_df['feature'] = feature_names\n",
    "        imp_df['gain'] = model.feature_importances_\n",
    "        imp_df['fold'] = fold_ + 1\n",
    "        \n",
    "        importances = pd.concat([importances, imp_df], axis=0)\n",
    "        \n",
    "        gc.collect() \n",
    "    return models,models_1,importances "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "422\n",
      "Training until validation scores don't improve for 250 rounds.\n",
      "[50]\tvalid_0's l1: 27.6401\tvalid_1's l1: 27.8644\n",
      "[100]\tvalid_0's l1: 23.8974\tvalid_1's l1: 24.1098\n",
      "[150]\tvalid_0's l1: 21.2366\tvalid_1's l1: 21.481\n",
      "[200]\tvalid_0's l1: 19.3635\tvalid_1's l1: 19.6343\n",
      "[250]\tvalid_0's l1: 18.0437\tvalid_1's l1: 18.3464\n",
      "[300]\tvalid_0's l1: 17.1246\tvalid_1's l1: 17.4756\n",
      "[350]\tvalid_0's l1: 16.478\tvalid_1's l1: 16.8828\n",
      "[400]\tvalid_0's l1: 16.0189\tvalid_1's l1: 16.4667\n",
      "[450]\tvalid_0's l1: 15.6864\tvalid_1's l1: 16.1697\n",
      "[500]\tvalid_0's l1: 15.4399\tvalid_1's l1: 15.9506\n",
      "[550]\tvalid_0's l1: 15.2488\tvalid_1's l1: 15.783\n",
      "[600]\tvalid_0's l1: 15.0962\tvalid_1's l1: 15.653\n",
      "[650]\tvalid_0's l1: 14.9725\tvalid_1's l1: 15.5524\n",
      "[700]\tvalid_0's l1: 14.8674\tvalid_1's l1: 15.4702\n",
      "[750]\tvalid_0's l1: 14.7777\tvalid_1's l1: 15.4038\n",
      "[800]\tvalid_0's l1: 14.6983\tvalid_1's l1: 15.3473\n",
      "[850]\tvalid_0's l1: 14.6263\tvalid_1's l1: 15.2988\n",
      "[900]\tvalid_0's l1: 14.5648\tvalid_1's l1: 15.2578\n",
      "[950]\tvalid_0's l1: 14.5098\tvalid_1's l1: 15.2253\n",
      "[1000]\tvalid_0's l1: 14.4598\tvalid_1's l1: 15.1977\n",
      "[1050]\tvalid_0's l1: 14.4127\tvalid_1's l1: 15.1728\n",
      "[1100]\tvalid_0's l1: 14.3702\tvalid_1's l1: 15.1518\n",
      "[1150]\tvalid_0's l1: 14.3298\tvalid_1's l1: 15.134\n",
      "[1200]\tvalid_0's l1: 14.2912\tvalid_1's l1: 15.116\n",
      "[1250]\tvalid_0's l1: 14.2559\tvalid_1's l1: 15.1013\n",
      "[1300]\tvalid_0's l1: 14.2213\tvalid_1's l1: 15.0872\n",
      "[1350]\tvalid_0's l1: 14.1899\tvalid_1's l1: 15.0767\n",
      "[1400]\tvalid_0's l1: 14.1581\tvalid_1's l1: 15.0659\n",
      "[1450]\tvalid_0's l1: 14.1283\tvalid_1's l1: 15.0568\n",
      "[1500]\tvalid_0's l1: 14.0975\tvalid_1's l1: 15.0476\n",
      "[1550]\tvalid_0's l1: 14.0701\tvalid_1's l1: 15.0402\n",
      "[1600]\tvalid_0's l1: 14.0451\tvalid_1's l1: 15.0355\n",
      "[1650]\tvalid_0's l1: 14.0205\tvalid_1's l1: 15.0305\n",
      "[1700]\tvalid_0's l1: 13.9949\tvalid_1's l1: 15.0244\n",
      "[1750]\tvalid_0's l1: 13.9715\tvalid_1's l1: 15.0208\n",
      "[1800]\tvalid_0's l1: 13.9482\tvalid_1's l1: 15.0163\n",
      "[1850]\tvalid_0's l1: 13.9244\tvalid_1's l1: 15.0117\n",
      "[1900]\tvalid_0's l1: 13.9026\tvalid_1's l1: 15.0088\n",
      "[1950]\tvalid_0's l1: 13.8793\tvalid_1's l1: 15.0037\n",
      "[2000]\tvalid_0's l1: 13.8572\tvalid_1's l1: 14.9995\n",
      "[2050]\tvalid_0's l1: 13.8362\tvalid_1's l1: 14.9967\n",
      "[2100]\tvalid_0's l1: 13.8145\tvalid_1's l1: 14.9932\n",
      "[2150]\tvalid_0's l1: 13.7933\tvalid_1's l1: 14.9901\n",
      "[2200]\tvalid_0's l1: 13.7734\tvalid_1's l1: 14.9877\n",
      "[2250]\tvalid_0's l1: 13.7534\tvalid_1's l1: 14.9853\n",
      "[2300]\tvalid_0's l1: 13.7338\tvalid_1's l1: 14.9834\n",
      "[2350]\tvalid_0's l1: 13.7141\tvalid_1's l1: 14.981\n",
      "[2400]\tvalid_0's l1: 13.6946\tvalid_1's l1: 14.979\n",
      "[2450]\tvalid_0's l1: 13.6762\tvalid_1's l1: 14.9762\n",
      "[2500]\tvalid_0's l1: 13.6567\tvalid_1's l1: 14.9749\n",
      "[2550]\tvalid_0's l1: 13.6384\tvalid_1's l1: 14.9728\n",
      "[2600]\tvalid_0's l1: 13.6194\tvalid_1's l1: 14.9702\n",
      "[2650]\tvalid_0's l1: 13.5997\tvalid_1's l1: 14.9668\n",
      "[2700]\tvalid_0's l1: 13.5811\tvalid_1's l1: 14.9646\n",
      "[2750]\tvalid_0's l1: 13.5638\tvalid_1's l1: 14.9633\n",
      "[2800]\tvalid_0's l1: 13.5447\tvalid_1's l1: 14.96\n",
      "[2850]\tvalid_0's l1: 13.5267\tvalid_1's l1: 14.9579\n",
      "[2900]\tvalid_0's l1: 13.5091\tvalid_1's l1: 14.9555\n",
      "[2950]\tvalid_0's l1: 13.4919\tvalid_1's l1: 14.9543\n",
      "[3000]\tvalid_0's l1: 13.4744\tvalid_1's l1: 14.9525\n",
      "[3050]\tvalid_0's l1: 13.4575\tvalid_1's l1: 14.9512\n",
      "[3100]\tvalid_0's l1: 13.4409\tvalid_1's l1: 14.9499\n",
      "[3150]\tvalid_0's l1: 13.4238\tvalid_1's l1: 14.9487\n",
      "[3200]\tvalid_0's l1: 13.4076\tvalid_1's l1: 14.9471\n",
      "[3250]\tvalid_0's l1: 13.3913\tvalid_1's l1: 14.9459\n",
      "[3300]\tvalid_0's l1: 13.3746\tvalid_1's l1: 14.9449\n",
      "[3350]\tvalid_0's l1: 13.3591\tvalid_1's l1: 14.9436\n",
      "[3400]\tvalid_0's l1: 13.3435\tvalid_1's l1: 14.9418\n",
      "[3450]\tvalid_0's l1: 13.3284\tvalid_1's l1: 14.941\n",
      "[3500]\tvalid_0's l1: 13.3137\tvalid_1's l1: 14.9402\n",
      "[3550]\tvalid_0's l1: 13.2979\tvalid_1's l1: 14.9388\n",
      "[3600]\tvalid_0's l1: 13.2829\tvalid_1's l1: 14.938\n",
      "[3650]\tvalid_0's l1: 13.2668\tvalid_1's l1: 14.9367\n",
      "[3700]\tvalid_0's l1: 13.2517\tvalid_1's l1: 14.9361\n",
      "[3750]\tvalid_0's l1: 13.2371\tvalid_1's l1: 14.9351\n",
      "[3800]\tvalid_0's l1: 13.2222\tvalid_1's l1: 14.9333\n",
      "[3850]\tvalid_0's l1: 13.2084\tvalid_1's l1: 14.9326\n",
      "[3900]\tvalid_0's l1: 13.1937\tvalid_1's l1: 14.9324\n",
      "[3950]\tvalid_0's l1: 13.1786\tvalid_1's l1: 14.9321\n",
      "[4000]\tvalid_0's l1: 13.1648\tvalid_1's l1: 14.9315\n",
      "[4050]\tvalid_0's l1: 13.15\tvalid_1's l1: 14.9301\n",
      "[4100]\tvalid_0's l1: 13.136\tvalid_1's l1: 14.9293\n",
      "[4150]\tvalid_0's l1: 13.1223\tvalid_1's l1: 14.9293\n",
      "[4200]\tvalid_0's l1: 13.1085\tvalid_1's l1: 14.9285\n",
      "[4250]\tvalid_0's l1: 13.0948\tvalid_1's l1: 14.9273\n",
      "[4300]\tvalid_0's l1: 13.0808\tvalid_1's l1: 14.9266\n",
      "[4350]\tvalid_0's l1: 13.0676\tvalid_1's l1: 14.9257\n",
      "[4400]\tvalid_0's l1: 13.0551\tvalid_1's l1: 14.9251\n",
      "[4450]\tvalid_0's l1: 13.0424\tvalid_1's l1: 14.9244\n",
      "[4500]\tvalid_0's l1: 13.0298\tvalid_1's l1: 14.924\n",
      "[4550]\tvalid_0's l1: 13.0166\tvalid_1's l1: 14.9239\n",
      "[4600]\tvalid_0's l1: 13.0032\tvalid_1's l1: 14.9234\n",
      "[4650]\tvalid_0's l1: 12.9902\tvalid_1's l1: 14.9225\n",
      "[4700]\tvalid_0's l1: 12.9772\tvalid_1's l1: 14.922\n",
      "[4750]\tvalid_0's l1: 12.9646\tvalid_1's l1: 14.9214\n",
      "[4800]\tvalid_0's l1: 12.9525\tvalid_1's l1: 14.921\n",
      "[4850]\tvalid_0's l1: 12.9399\tvalid_1's l1: 14.9205\n",
      "[4900]\tvalid_0's l1: 12.9277\tvalid_1's l1: 14.9201\n",
      "[4950]\tvalid_0's l1: 12.9162\tvalid_1's l1: 14.92\n",
      "[5000]\tvalid_0's l1: 12.9032\tvalid_1's l1: 14.9194\n",
      "[5050]\tvalid_0's l1: 12.8908\tvalid_1's l1: 14.9192\n",
      "[5100]\tvalid_0's l1: 12.8785\tvalid_1's l1: 14.9189\n",
      "[5150]\tvalid_0's l1: 12.8664\tvalid_1's l1: 14.9186\n",
      "[5200]\tvalid_0's l1: 12.8538\tvalid_1's l1: 14.9178\n",
      "[5250]\tvalid_0's l1: 12.841\tvalid_1's l1: 14.9169\n",
      "[5300]\tvalid_0's l1: 12.8302\tvalid_1's l1: 14.9167\n",
      "[5350]\tvalid_0's l1: 12.819\tvalid_1's l1: 14.9163\n",
      "[5400]\tvalid_0's l1: 12.8077\tvalid_1's l1: 14.9161\n",
      "[5450]\tvalid_0's l1: 12.7963\tvalid_1's l1: 14.9157\n",
      "[5500]\tvalid_0's l1: 12.7846\tvalid_1's l1: 14.9155\n",
      "[5550]\tvalid_0's l1: 12.7733\tvalid_1's l1: 14.9146\n",
      "[5600]\tvalid_0's l1: 12.7625\tvalid_1's l1: 14.9145\n",
      "[5650]\tvalid_0's l1: 12.7512\tvalid_1's l1: 14.914\n",
      "[5700]\tvalid_0's l1: 12.7401\tvalid_1's l1: 14.9136\n",
      "[5750]\tvalid_0's l1: 12.7298\tvalid_1's l1: 14.9133\n",
      "[5800]\tvalid_0's l1: 12.72\tvalid_1's l1: 14.9131\n",
      "[5850]\tvalid_0's l1: 12.7093\tvalid_1's l1: 14.9129\n",
      "[5900]\tvalid_0's l1: 12.6977\tvalid_1's l1: 14.912\n",
      "[5950]\tvalid_0's l1: 12.6868\tvalid_1's l1: 14.9118\n",
      "[6000]\tvalid_0's l1: 12.6763\tvalid_1's l1: 14.9113\n",
      "[6050]\tvalid_0's l1: 12.6658\tvalid_1's l1: 14.911\n",
      "[6100]\tvalid_0's l1: 12.656\tvalid_1's l1: 14.9105\n",
      "[6150]\tvalid_0's l1: 12.6451\tvalid_1's l1: 14.9098\n",
      "[6200]\tvalid_0's l1: 12.6338\tvalid_1's l1: 14.909\n",
      "[6250]\tvalid_0's l1: 12.6223\tvalid_1's l1: 14.9083\n",
      "[6300]\tvalid_0's l1: 12.6126\tvalid_1's l1: 14.9081\n",
      "[6350]\tvalid_0's l1: 12.6022\tvalid_1's l1: 14.908\n",
      "[6400]\tvalid_0's l1: 12.5923\tvalid_1's l1: 14.9079\n",
      "[6450]\tvalid_0's l1: 12.5819\tvalid_1's l1: 14.9076\n",
      "[6500]\tvalid_0's l1: 12.571\tvalid_1's l1: 14.9075\n",
      "[6550]\tvalid_0's l1: 12.5619\tvalid_1's l1: 14.9073\n",
      "[6600]\tvalid_0's l1: 12.5525\tvalid_1's l1: 14.9074\n",
      "[6650]\tvalid_0's l1: 12.5445\tvalid_1's l1: 14.9075\n",
      "[6700]\tvalid_0's l1: 12.5359\tvalid_1's l1: 14.9072\n",
      "[6750]\tvalid_0's l1: 12.5279\tvalid_1's l1: 14.9069\n",
      "[6800]\tvalid_0's l1: 12.5204\tvalid_1's l1: 14.9069\n",
      "[6850]\tvalid_0's l1: 12.5133\tvalid_1's l1: 14.9067\n",
      "[6900]\tvalid_0's l1: 12.5046\tvalid_1's l1: 14.9068\n",
      "[6950]\tvalid_0's l1: 12.4964\tvalid_1's l1: 14.9066\n",
      "[7000]\tvalid_0's l1: 12.4863\tvalid_1's l1: 14.9065\n",
      "[7050]\tvalid_0's l1: 12.4787\tvalid_1's l1: 14.9064\n",
      "[7100]\tvalid_0's l1: 12.4715\tvalid_1's l1: 14.9065\n",
      "[7150]\tvalid_0's l1: 12.4647\tvalid_1's l1: 14.9067\n",
      "[7200]\tvalid_0's l1: 12.459\tvalid_1's l1: 14.9066\n",
      "[7250]\tvalid_0's l1: 12.4533\tvalid_1's l1: 14.9062\n",
      "[7300]\tvalid_0's l1: 12.4482\tvalid_1's l1: 14.906\n",
      "[7350]\tvalid_0's l1: 12.4437\tvalid_1's l1: 14.906\n",
      "[7400]\tvalid_0's l1: 12.4387\tvalid_1's l1: 14.906\n",
      "[7450]\tvalid_0's l1: 12.4331\tvalid_1's l1: 14.9063\n",
      "[7500]\tvalid_0's l1: 12.4275\tvalid_1's l1: 14.9062\n",
      "[7550]\tvalid_0's l1: 12.422\tvalid_1's l1: 14.9058\n",
      "[7600]\tvalid_0's l1: 12.4156\tvalid_1's l1: 14.9058\n",
      "[7650]\tvalid_0's l1: 12.4089\tvalid_1's l1: 14.9057\n",
      "[7700]\tvalid_0's l1: 12.4042\tvalid_1's l1: 14.9058\n",
      "[7750]\tvalid_0's l1: 12.3982\tvalid_1's l1: 14.9061\n",
      "[7800]\tvalid_0's l1: 12.3905\tvalid_1's l1: 14.9058\n",
      "[7850]\tvalid_0's l1: 12.3817\tvalid_1's l1: 14.9057\n",
      "[7900]\tvalid_0's l1: 12.3731\tvalid_1's l1: 14.9056\n",
      "[7950]\tvalid_0's l1: 12.3641\tvalid_1's l1: 14.9061\n",
      "[8000]\tvalid_0's l1: 12.3576\tvalid_1's l1: 14.9063\n",
      "[8050]\tvalid_0's l1: 12.3509\tvalid_1's l1: 14.907\n",
      "[8100]\tvalid_0's l1: 12.3446\tvalid_1's l1: 14.9072\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Early stopping, best iteration is:\n",
      "[7882]\tvalid_0's l1: 12.3762\tvalid_1's l1: 14.9055\n",
      "Training until validation scores don't improve for 250 rounds.\n",
      "[50]\tvalid_0's l1: 28.1121\tvalid_1's l1: 28.3035\n",
      "[100]\tvalid_0's l1: 24.1254\tvalid_1's l1: 24.3496\n",
      "[150]\tvalid_0's l1: 21.2909\tvalid_1's l1: 21.5635\n",
      "[200]\tvalid_0's l1: 19.3152\tvalid_1's l1: 19.6053\n",
      "[250]\tvalid_0's l1: 17.9673\tvalid_1's l1: 18.276\n",
      "[300]\tvalid_0's l1: 17.0483\tvalid_1's l1: 17.3863\n",
      "[350]\tvalid_0's l1: 16.4168\tvalid_1's l1: 16.7916\n",
      "[400]\tvalid_0's l1: 15.9745\tvalid_1's l1: 16.3805\n",
      "[450]\tvalid_0's l1: 15.6597\tvalid_1's l1: 16.0976\n",
      "[500]\tvalid_0's l1: 15.4273\tvalid_1's l1: 15.8955\n",
      "[550]\tvalid_0's l1: 15.2502\tvalid_1's l1: 15.7401\n",
      "[600]\tvalid_0's l1: 15.1103\tvalid_1's l1: 15.624\n",
      "[650]\tvalid_0's l1: 14.9953\tvalid_1's l1: 15.532\n",
      "[700]\tvalid_0's l1: 14.9014\tvalid_1's l1: 15.4591\n",
      "[750]\tvalid_0's l1: 14.8181\tvalid_1's l1: 15.3942\n",
      "[800]\tvalid_0's l1: 14.7436\tvalid_1's l1: 15.3366\n",
      "[850]\tvalid_0's l1: 14.6789\tvalid_1's l1: 15.2896\n",
      "[900]\tvalid_0's l1: 14.6222\tvalid_1's l1: 15.2533\n",
      "[950]\tvalid_0's l1: 14.5698\tvalid_1's l1: 15.2205\n",
      "[1000]\tvalid_0's l1: 14.5197\tvalid_1's l1: 15.1893\n",
      "[1050]\tvalid_0's l1: 14.4736\tvalid_1's l1: 15.1627\n",
      "[1100]\tvalid_0's l1: 14.4312\tvalid_1's l1: 15.1383\n",
      "[1150]\tvalid_0's l1: 14.3885\tvalid_1's l1: 15.1148\n",
      "[1200]\tvalid_0's l1: 14.3504\tvalid_1's l1: 15.0946\n",
      "[1250]\tvalid_0's l1: 14.3152\tvalid_1's l1: 15.0784\n",
      "[1300]\tvalid_0's l1: 14.2816\tvalid_1's l1: 15.0638\n",
      "[1350]\tvalid_0's l1: 14.2495\tvalid_1's l1: 15.0515\n",
      "[1400]\tvalid_0's l1: 14.2185\tvalid_1's l1: 15.0404\n",
      "[1450]\tvalid_0's l1: 14.189\tvalid_1's l1: 15.0303\n",
      "[1500]\tvalid_0's l1: 14.1607\tvalid_1's l1: 15.0213\n",
      "[1550]\tvalid_0's l1: 14.1321\tvalid_1's l1: 15.0116\n",
      "[1600]\tvalid_0's l1: 14.1056\tvalid_1's l1: 15.0043\n",
      "[1650]\tvalid_0's l1: 14.0782\tvalid_1's l1: 14.9957\n",
      "[1700]\tvalid_0's l1: 14.0532\tvalid_1's l1: 14.9908\n",
      "[1750]\tvalid_0's l1: 14.0288\tvalid_1's l1: 14.9855\n",
      "[1800]\tvalid_0's l1: 14.0046\tvalid_1's l1: 14.9795\n",
      "[1850]\tvalid_0's l1: 13.9811\tvalid_1's l1: 14.9745\n",
      "[1900]\tvalid_0's l1: 13.958\tvalid_1's l1: 14.9702\n",
      "[1950]\tvalid_0's l1: 13.9348\tvalid_1's l1: 14.9648\n",
      "[2000]\tvalid_0's l1: 13.912\tvalid_1's l1: 14.9606\n",
      "[2050]\tvalid_0's l1: 13.8909\tvalid_1's l1: 14.9576\n",
      "[2100]\tvalid_0's l1: 13.87\tvalid_1's l1: 14.9546\n",
      "[2150]\tvalid_0's l1: 13.8494\tvalid_1's l1: 14.9527\n",
      "[2200]\tvalid_0's l1: 13.8281\tvalid_1's l1: 14.9495\n",
      "[2250]\tvalid_0's l1: 13.8077\tvalid_1's l1: 14.9465\n",
      "[2300]\tvalid_0's l1: 13.7871\tvalid_1's l1: 14.943\n",
      "[2350]\tvalid_0's l1: 13.7667\tvalid_1's l1: 14.9411\n",
      "[2400]\tvalid_0's l1: 13.7459\tvalid_1's l1: 14.9374\n",
      "[2450]\tvalid_0's l1: 13.7268\tvalid_1's l1: 14.9355\n",
      "[2500]\tvalid_0's l1: 13.7073\tvalid_1's l1: 14.9323\n",
      "[2550]\tvalid_0's l1: 13.6884\tvalid_1's l1: 14.9299\n",
      "[2600]\tvalid_0's l1: 13.6696\tvalid_1's l1: 14.928\n",
      "[2650]\tvalid_0's l1: 13.6507\tvalid_1's l1: 14.9263\n",
      "[2700]\tvalid_0's l1: 13.6326\tvalid_1's l1: 14.9245\n",
      "[2750]\tvalid_0's l1: 13.6139\tvalid_1's l1: 14.922\n",
      "[2800]\tvalid_0's l1: 13.5955\tvalid_1's l1: 14.9203\n",
      "[2850]\tvalid_0's l1: 13.5775\tvalid_1's l1: 14.9188\n",
      "[2900]\tvalid_0's l1: 13.5595\tvalid_1's l1: 14.9172\n",
      "[2950]\tvalid_0's l1: 13.5419\tvalid_1's l1: 14.9157\n",
      "[3000]\tvalid_0's l1: 13.5241\tvalid_1's l1: 14.9148\n",
      "[3050]\tvalid_0's l1: 13.5061\tvalid_1's l1: 14.9126\n",
      "[3100]\tvalid_0's l1: 13.4889\tvalid_1's l1: 14.9112\n",
      "[3150]\tvalid_0's l1: 13.4717\tvalid_1's l1: 14.9093\n",
      "[3200]\tvalid_0's l1: 13.4546\tvalid_1's l1: 14.9079\n",
      "[3250]\tvalid_0's l1: 13.4372\tvalid_1's l1: 14.9066\n",
      "[3300]\tvalid_0's l1: 13.42\tvalid_1's l1: 14.9049\n",
      "[3350]\tvalid_0's l1: 13.403\tvalid_1's l1: 14.9034\n",
      "[3400]\tvalid_0's l1: 13.3861\tvalid_1's l1: 14.9018\n",
      "[3450]\tvalid_0's l1: 13.3691\tvalid_1's l1: 14.9005\n",
      "[3500]\tvalid_0's l1: 13.3522\tvalid_1's l1: 14.8994\n",
      "[3550]\tvalid_0's l1: 13.3354\tvalid_1's l1: 14.8993\n",
      "[3600]\tvalid_0's l1: 13.3186\tvalid_1's l1: 14.8985\n",
      "[3650]\tvalid_0's l1: 13.3021\tvalid_1's l1: 14.8973\n",
      "[3700]\tvalid_0's l1: 13.2861\tvalid_1's l1: 14.8969\n",
      "[3750]\tvalid_0's l1: 13.2706\tvalid_1's l1: 14.8962\n",
      "[3800]\tvalid_0's l1: 13.2546\tvalid_1's l1: 14.8957\n",
      "[3850]\tvalid_0's l1: 13.2384\tvalid_1's l1: 14.895\n",
      "[3900]\tvalid_0's l1: 13.2223\tvalid_1's l1: 14.8942\n",
      "[3950]\tvalid_0's l1: 13.2058\tvalid_1's l1: 14.8946\n",
      "[4000]\tvalid_0's l1: 13.1893\tvalid_1's l1: 14.8946\n",
      "[4050]\tvalid_0's l1: 13.1738\tvalid_1's l1: 14.8944\n",
      "[4100]\tvalid_0's l1: 13.1573\tvalid_1's l1: 14.8939\n",
      "[4150]\tvalid_0's l1: 13.1412\tvalid_1's l1: 14.8939\n",
      "[4200]\tvalid_0's l1: 13.1254\tvalid_1's l1: 14.8934\n",
      "[4250]\tvalid_0's l1: 13.1089\tvalid_1's l1: 14.8921\n",
      "[4300]\tvalid_0's l1: 13.0931\tvalid_1's l1: 14.8922\n",
      "[4350]\tvalid_0's l1: 13.0776\tvalid_1's l1: 14.8915\n",
      "[4400]\tvalid_0's l1: 13.0622\tvalid_1's l1: 14.891\n",
      "[4450]\tvalid_0's l1: 13.0466\tvalid_1's l1: 14.8903\n",
      "[4500]\tvalid_0's l1: 13.0315\tvalid_1's l1: 14.8898\n",
      "[4550]\tvalid_0's l1: 13.0161\tvalid_1's l1: 14.8894\n",
      "[4600]\tvalid_0's l1: 13.0006\tvalid_1's l1: 14.89\n",
      "[4650]\tvalid_0's l1: 12.985\tvalid_1's l1: 14.8898\n",
      "[4700]\tvalid_0's l1: 12.97\tvalid_1's l1: 14.8904\n",
      "[4750]\tvalid_0's l1: 12.9538\tvalid_1's l1: 14.8901\n",
      "[4800]\tvalid_0's l1: 12.9386\tvalid_1's l1: 14.8896\n",
      "Early stopping, best iteration is:\n",
      "[4568]\tvalid_0's l1: 13.0102\tvalid_1's l1: 14.8892\n",
      "****************************************************************************************************\n",
      "MAE Model 0.06287124227830537\n",
      "MSE Model 0.06293565921970032\n",
      "Merge Model12 0.063039062547442\n",
      "Training until validation scores don't improve for 250 rounds.\n",
      "[50]\tvalid_0's l1: 27.6629\tvalid_1's l1: 27.9519\n",
      "[100]\tvalid_0's l1: 23.9261\tvalid_1's l1: 24.2437\n",
      "[150]\tvalid_0's l1: 21.2602\tvalid_1's l1: 21.5553\n",
      "[200]\tvalid_0's l1: 19.387\tvalid_1's l1: 19.6451\n",
      "[250]\tvalid_0's l1: 18.0781\tvalid_1's l1: 18.2949\n",
      "[300]\tvalid_0's l1: 17.1647\tvalid_1's l1: 17.3578\n",
      "[350]\tvalid_0's l1: 16.5231\tvalid_1's l1: 16.71\n",
      "[400]\tvalid_0's l1: 16.0609\tvalid_1's l1: 16.2601\n",
      "[450]\tvalid_0's l1: 15.7255\tvalid_1's l1: 15.9429\n",
      "[500]\tvalid_0's l1: 15.4744\tvalid_1's l1: 15.7112\n",
      "[550]\tvalid_0's l1: 15.282\tvalid_1's l1: 15.545\n",
      "[600]\tvalid_0's l1: 15.1275\tvalid_1's l1: 15.4176\n",
      "[650]\tvalid_0's l1: 15.0047\tvalid_1's l1: 15.3209\n",
      "[700]\tvalid_0's l1: 14.8979\tvalid_1's l1: 15.2422\n",
      "[750]\tvalid_0's l1: 14.8067\tvalid_1's l1: 15.1797\n",
      "[800]\tvalid_0's l1: 14.729\tvalid_1's l1: 15.1287\n",
      "[850]\tvalid_0's l1: 14.6595\tvalid_1's l1: 15.0845\n",
      "[900]\tvalid_0's l1: 14.5989\tvalid_1's l1: 15.0508\n",
      "[950]\tvalid_0's l1: 14.5448\tvalid_1's l1: 15.0227\n",
      "[1000]\tvalid_0's l1: 14.4957\tvalid_1's l1: 14.9984\n",
      "[1050]\tvalid_0's l1: 14.4487\tvalid_1's l1: 14.9762\n",
      "[1100]\tvalid_0's l1: 14.4042\tvalid_1's l1: 14.9568\n",
      "[1150]\tvalid_0's l1: 14.3646\tvalid_1's l1: 14.9413\n",
      "[1200]\tvalid_0's l1: 14.3252\tvalid_1's l1: 14.9258\n",
      "[1250]\tvalid_0's l1: 14.2898\tvalid_1's l1: 14.9136\n",
      "[1300]\tvalid_0's l1: 14.2545\tvalid_1's l1: 14.9015\n",
      "[1350]\tvalid_0's l1: 14.2208\tvalid_1's l1: 14.8908\n",
      "[1400]\tvalid_0's l1: 14.1874\tvalid_1's l1: 14.8791\n",
      "[1450]\tvalid_0's l1: 14.1569\tvalid_1's l1: 14.8709\n",
      "[1500]\tvalid_0's l1: 14.1274\tvalid_1's l1: 14.8616\n",
      "[1550]\tvalid_0's l1: 14.099\tvalid_1's l1: 14.8545\n",
      "[1600]\tvalid_0's l1: 14.0719\tvalid_1's l1: 14.849\n",
      "[1650]\tvalid_0's l1: 14.0445\tvalid_1's l1: 14.842\n",
      "[1700]\tvalid_0's l1: 14.0191\tvalid_1's l1: 14.8383\n",
      "[1750]\tvalid_0's l1: 13.9942\tvalid_1's l1: 14.8325\n",
      "[1800]\tvalid_0's l1: 13.9701\tvalid_1's l1: 14.8271\n",
      "[1850]\tvalid_0's l1: 13.9478\tvalid_1's l1: 14.8238\n",
      "[1900]\tvalid_0's l1: 13.925\tvalid_1's l1: 14.8198\n",
      "[1950]\tvalid_0's l1: 13.9018\tvalid_1's l1: 14.8146\n",
      "[2000]\tvalid_0's l1: 13.8797\tvalid_1's l1: 14.8122\n",
      "[2050]\tvalid_0's l1: 13.8591\tvalid_1's l1: 14.811\n",
      "[2100]\tvalid_0's l1: 13.838\tvalid_1's l1: 14.8083\n",
      "[2150]\tvalid_0's l1: 13.8176\tvalid_1's l1: 14.805\n",
      "[2200]\tvalid_0's l1: 13.7963\tvalid_1's l1: 14.802\n",
      "[2250]\tvalid_0's l1: 13.7764\tvalid_1's l1: 14.7991\n",
      "[2300]\tvalid_0's l1: 13.756\tvalid_1's l1: 14.7958\n",
      "[2350]\tvalid_0's l1: 13.7371\tvalid_1's l1: 14.7934\n",
      "[2400]\tvalid_0's l1: 13.7181\tvalid_1's l1: 14.7913\n",
      "[2450]\tvalid_0's l1: 13.6989\tvalid_1's l1: 14.7887\n",
      "[2500]\tvalid_0's l1: 13.6805\tvalid_1's l1: 14.7871\n",
      "[2550]\tvalid_0's l1: 13.6618\tvalid_1's l1: 14.7853\n",
      "[2600]\tvalid_0's l1: 13.6434\tvalid_1's l1: 14.7831\n",
      "[2650]\tvalid_0's l1: 13.625\tvalid_1's l1: 14.7812\n",
      "[2700]\tvalid_0's l1: 13.6074\tvalid_1's l1: 14.7804\n",
      "[2750]\tvalid_0's l1: 13.589\tvalid_1's l1: 14.7793\n",
      "[2800]\tvalid_0's l1: 13.5722\tvalid_1's l1: 14.778\n",
      "[2850]\tvalid_0's l1: 13.5552\tvalid_1's l1: 14.7762\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2900]\tvalid_0's l1: 13.5369\tvalid_1's l1: 14.7742\n",
      "[2950]\tvalid_0's l1: 13.5201\tvalid_1's l1: 14.7727\n",
      "[3000]\tvalid_0's l1: 13.503\tvalid_1's l1: 14.7714\n",
      "[3050]\tvalid_0's l1: 13.4873\tvalid_1's l1: 14.7711\n",
      "[3100]\tvalid_0's l1: 13.4708\tvalid_1's l1: 14.7696\n",
      "[3150]\tvalid_0's l1: 13.454\tvalid_1's l1: 14.7692\n",
      "[3200]\tvalid_0's l1: 13.4379\tvalid_1's l1: 14.7685\n",
      "[3250]\tvalid_0's l1: 13.422\tvalid_1's l1: 14.7669\n",
      "[3300]\tvalid_0's l1: 13.4062\tvalid_1's l1: 14.7661\n",
      "[3350]\tvalid_0's l1: 13.3907\tvalid_1's l1: 14.7647\n",
      "[3400]\tvalid_0's l1: 13.3738\tvalid_1's l1: 14.7631\n",
      "[3450]\tvalid_0's l1: 13.3583\tvalid_1's l1: 14.7619\n",
      "[3500]\tvalid_0's l1: 13.3418\tvalid_1's l1: 14.7607\n",
      "[3550]\tvalid_0's l1: 13.3263\tvalid_1's l1: 14.7588\n",
      "[3600]\tvalid_0's l1: 13.3107\tvalid_1's l1: 14.7579\n",
      "[3650]\tvalid_0's l1: 13.2956\tvalid_1's l1: 14.7563\n",
      "[3700]\tvalid_0's l1: 13.2802\tvalid_1's l1: 14.7549\n",
      "[3750]\tvalid_0's l1: 13.2645\tvalid_1's l1: 14.7539\n",
      "[3800]\tvalid_0's l1: 13.25\tvalid_1's l1: 14.7533\n",
      "[3850]\tvalid_0's l1: 13.2355\tvalid_1's l1: 14.7528\n",
      "[3900]\tvalid_0's l1: 13.2202\tvalid_1's l1: 14.7523\n",
      "[3950]\tvalid_0's l1: 13.2052\tvalid_1's l1: 14.7511\n",
      "[4000]\tvalid_0's l1: 13.1907\tvalid_1's l1: 14.7506\n",
      "[4050]\tvalid_0's l1: 13.1762\tvalid_1's l1: 14.7494\n",
      "[4100]\tvalid_0's l1: 13.1615\tvalid_1's l1: 14.7486\n",
      "[4150]\tvalid_0's l1: 13.147\tvalid_1's l1: 14.7476\n",
      "[4200]\tvalid_0's l1: 13.1326\tvalid_1's l1: 14.7467\n",
      "[4250]\tvalid_0's l1: 13.1193\tvalid_1's l1: 14.7469\n",
      "[4300]\tvalid_0's l1: 13.1054\tvalid_1's l1: 14.7463\n",
      "[4350]\tvalid_0's l1: 13.0914\tvalid_1's l1: 14.7454\n",
      "[4400]\tvalid_0's l1: 13.0778\tvalid_1's l1: 14.7447\n",
      "[4450]\tvalid_0's l1: 13.0647\tvalid_1's l1: 14.7445\n",
      "[4500]\tvalid_0's l1: 13.0517\tvalid_1's l1: 14.7436\n",
      "[4550]\tvalid_0's l1: 13.04\tvalid_1's l1: 14.7436\n",
      "[4600]\tvalid_0's l1: 13.027\tvalid_1's l1: 14.7431\n",
      "[4650]\tvalid_0's l1: 13.0136\tvalid_1's l1: 14.7427\n",
      "[4700]\tvalid_0's l1: 13.001\tvalid_1's l1: 14.7419\n",
      "[4750]\tvalid_0's l1: 12.989\tvalid_1's l1: 14.7418\n",
      "[4800]\tvalid_0's l1: 12.9765\tvalid_1's l1: 14.7412\n",
      "[4850]\tvalid_0's l1: 12.9656\tvalid_1's l1: 14.741\n",
      "[4900]\tvalid_0's l1: 12.9549\tvalid_1's l1: 14.7409\n",
      "[4950]\tvalid_0's l1: 12.9435\tvalid_1's l1: 14.7408\n",
      "[5000]\tvalid_0's l1: 12.9312\tvalid_1's l1: 14.7408\n",
      "[5050]\tvalid_0's l1: 12.919\tvalid_1's l1: 14.7404\n",
      "[5100]\tvalid_0's l1: 12.9073\tvalid_1's l1: 14.7402\n",
      "[5150]\tvalid_0's l1: 12.8959\tvalid_1's l1: 14.7398\n",
      "[5200]\tvalid_0's l1: 12.8839\tvalid_1's l1: 14.7395\n",
      "[5250]\tvalid_0's l1: 12.8716\tvalid_1's l1: 14.7392\n",
      "[5300]\tvalid_0's l1: 12.8601\tvalid_1's l1: 14.739\n",
      "[5350]\tvalid_0's l1: 12.8477\tvalid_1's l1: 14.7385\n",
      "[5400]\tvalid_0's l1: 12.8362\tvalid_1's l1: 14.7384\n",
      "[5450]\tvalid_0's l1: 12.8252\tvalid_1's l1: 14.7382\n",
      "[5500]\tvalid_0's l1: 12.8144\tvalid_1's l1: 14.7384\n",
      "[5550]\tvalid_0's l1: 12.8024\tvalid_1's l1: 14.7383\n",
      "[5600]\tvalid_0's l1: 12.7908\tvalid_1's l1: 14.7378\n",
      "[5650]\tvalid_0's l1: 12.7798\tvalid_1's l1: 14.7373\n",
      "[5700]\tvalid_0's l1: 12.7677\tvalid_1's l1: 14.7368\n",
      "[5750]\tvalid_0's l1: 12.7563\tvalid_1's l1: 14.7362\n",
      "[5800]\tvalid_0's l1: 12.7453\tvalid_1's l1: 14.7353\n",
      "[5850]\tvalid_0's l1: 12.7341\tvalid_1's l1: 14.7348\n",
      "[5900]\tvalid_0's l1: 12.7232\tvalid_1's l1: 14.7347\n",
      "[5950]\tvalid_0's l1: 12.7125\tvalid_1's l1: 14.7345\n",
      "[6000]\tvalid_0's l1: 12.7016\tvalid_1's l1: 14.7343\n",
      "[6050]\tvalid_0's l1: 12.692\tvalid_1's l1: 14.7343\n",
      "[6100]\tvalid_0's l1: 12.6837\tvalid_1's l1: 14.7344\n",
      "[6150]\tvalid_0's l1: 12.6736\tvalid_1's l1: 14.7338\n",
      "[6200]\tvalid_0's l1: 12.6629\tvalid_1's l1: 14.7339\n",
      "[6250]\tvalid_0's l1: 12.6534\tvalid_1's l1: 14.7336\n",
      "[6300]\tvalid_0's l1: 12.643\tvalid_1's l1: 14.7329\n",
      "[6350]\tvalid_0's l1: 12.6321\tvalid_1's l1: 14.7326\n",
      "[6400]\tvalid_0's l1: 12.6223\tvalid_1's l1: 14.7327\n",
      "[6450]\tvalid_0's l1: 12.6122\tvalid_1's l1: 14.7324\n",
      "[6500]\tvalid_0's l1: 12.6026\tvalid_1's l1: 14.7325\n",
      "[6550]\tvalid_0's l1: 12.5945\tvalid_1's l1: 14.7327\n",
      "[6600]\tvalid_0's l1: 12.5842\tvalid_1's l1: 14.7329\n",
      "[6650]\tvalid_0's l1: 12.5742\tvalid_1's l1: 14.7325\n",
      "Early stopping, best iteration is:\n",
      "[6446]\tvalid_0's l1: 12.613\tvalid_1's l1: 14.7323\n",
      "Training until validation scores don't improve for 250 rounds.\n",
      "[50]\tvalid_0's l1: 28.1182\tvalid_1's l1: 28.2833\n",
      "[100]\tvalid_0's l1: 24.1315\tvalid_1's l1: 24.2879\n",
      "[150]\tvalid_0's l1: 21.3026\tvalid_1's l1: 21.4367\n",
      "[200]\tvalid_0's l1: 19.326\tvalid_1's l1: 19.4726\n",
      "[250]\tvalid_0's l1: 17.9689\tvalid_1's l1: 18.1275\n",
      "[300]\tvalid_0's l1: 17.0503\tvalid_1's l1: 17.2224\n",
      "[350]\tvalid_0's l1: 16.4231\tvalid_1's l1: 16.6033\n",
      "[400]\tvalid_0's l1: 15.9846\tvalid_1's l1: 16.1767\n",
      "[450]\tvalid_0's l1: 15.6711\tvalid_1's l1: 15.8798\n",
      "[500]\tvalid_0's l1: 15.44\tvalid_1's l1: 15.6669\n",
      "[550]\tvalid_0's l1: 15.2655\tvalid_1's l1: 15.506\n",
      "[600]\tvalid_0's l1: 15.1267\tvalid_1's l1: 15.3879\n",
      "[650]\tvalid_0's l1: 15.017\tvalid_1's l1: 15.3001\n",
      "[700]\tvalid_0's l1: 14.921\tvalid_1's l1: 15.227\n",
      "[750]\tvalid_0's l1: 14.8356\tvalid_1's l1: 15.1666\n",
      "[800]\tvalid_0's l1: 14.7599\tvalid_1's l1: 15.1159\n",
      "[850]\tvalid_0's l1: 14.6948\tvalid_1's l1: 15.075\n",
      "[900]\tvalid_0's l1: 14.637\tvalid_1's l1: 15.0401\n",
      "[950]\tvalid_0's l1: 14.5845\tvalid_1's l1: 15.0098\n",
      "[1000]\tvalid_0's l1: 14.536\tvalid_1's l1: 14.9833\n",
      "[1050]\tvalid_0's l1: 14.4902\tvalid_1's l1: 14.9601\n",
      "[1100]\tvalid_0's l1: 14.4474\tvalid_1's l1: 14.9397\n",
      "[1150]\tvalid_0's l1: 14.4063\tvalid_1's l1: 14.9205\n",
      "[1200]\tvalid_0's l1: 14.3685\tvalid_1's l1: 14.9037\n",
      "[1250]\tvalid_0's l1: 14.3337\tvalid_1's l1: 14.8893\n",
      "[1300]\tvalid_0's l1: 14.2993\tvalid_1's l1: 14.876\n",
      "[1350]\tvalid_0's l1: 14.266\tvalid_1's l1: 14.8644\n",
      "[1400]\tvalid_0's l1: 14.2352\tvalid_1's l1: 14.8538\n",
      "[1450]\tvalid_0's l1: 14.2057\tvalid_1's l1: 14.8458\n",
      "[1500]\tvalid_0's l1: 14.1768\tvalid_1's l1: 14.8376\n",
      "[1550]\tvalid_0's l1: 14.1481\tvalid_1's l1: 14.8291\n",
      "[1600]\tvalid_0's l1: 14.1216\tvalid_1's l1: 14.8228\n",
      "[1650]\tvalid_0's l1: 14.0954\tvalid_1's l1: 14.8175\n",
      "[1700]\tvalid_0's l1: 14.0701\tvalid_1's l1: 14.8125\n",
      "[1750]\tvalid_0's l1: 14.0456\tvalid_1's l1: 14.8084\n",
      "[1800]\tvalid_0's l1: 14.0211\tvalid_1's l1: 14.8045\n",
      "[1850]\tvalid_0's l1: 13.9982\tvalid_1's l1: 14.8018\n",
      "[1900]\tvalid_0's l1: 13.9758\tvalid_1's l1: 14.7991\n",
      "[1950]\tvalid_0's l1: 13.9533\tvalid_1's l1: 14.7963\n",
      "[2000]\tvalid_0's l1: 13.931\tvalid_1's l1: 14.7937\n",
      "[2050]\tvalid_0's l1: 13.91\tvalid_1's l1: 14.7935\n",
      "[2100]\tvalid_0's l1: 13.8888\tvalid_1's l1: 14.7916\n",
      "[2150]\tvalid_0's l1: 13.8683\tvalid_1's l1: 14.7904\n",
      "[2200]\tvalid_0's l1: 13.8465\tvalid_1's l1: 14.7889\n",
      "[2250]\tvalid_0's l1: 13.8267\tvalid_1's l1: 14.7885\n",
      "[2300]\tvalid_0's l1: 13.8058\tvalid_1's l1: 14.787\n",
      "[2350]\tvalid_0's l1: 13.7864\tvalid_1's l1: 14.7859\n",
      "[2400]\tvalid_0's l1: 13.7662\tvalid_1's l1: 14.7843\n",
      "[2450]\tvalid_0's l1: 13.7466\tvalid_1's l1: 14.7833\n",
      "[2500]\tvalid_0's l1: 13.7271\tvalid_1's l1: 14.7809\n",
      "[2550]\tvalid_0's l1: 13.7081\tvalid_1's l1: 14.7794\n",
      "[2600]\tvalid_0's l1: 13.6892\tvalid_1's l1: 14.7787\n",
      "[2650]\tvalid_0's l1: 13.6703\tvalid_1's l1: 14.7778\n",
      "[2700]\tvalid_0's l1: 13.6517\tvalid_1's l1: 14.7765\n",
      "[2750]\tvalid_0's l1: 13.6327\tvalid_1's l1: 14.7751\n",
      "[2800]\tvalid_0's l1: 13.6144\tvalid_1's l1: 14.7748\n",
      "[2850]\tvalid_0's l1: 13.5964\tvalid_1's l1: 14.7747\n",
      "[2900]\tvalid_0's l1: 13.5779\tvalid_1's l1: 14.7733\n",
      "[2950]\tvalid_0's l1: 13.5598\tvalid_1's l1: 14.7726\n",
      "[3000]\tvalid_0's l1: 13.5413\tvalid_1's l1: 14.7722\n",
      "[3050]\tvalid_0's l1: 13.5233\tvalid_1's l1: 14.7719\n",
      "[3100]\tvalid_0's l1: 13.5058\tvalid_1's l1: 14.7716\n",
      "[3150]\tvalid_0's l1: 13.4889\tvalid_1's l1: 14.7715\n",
      "[3200]\tvalid_0's l1: 13.4713\tvalid_1's l1: 14.7701\n",
      "[3250]\tvalid_0's l1: 13.4536\tvalid_1's l1: 14.7695\n",
      "[3300]\tvalid_0's l1: 13.4369\tvalid_1's l1: 14.7681\n",
      "[3350]\tvalid_0's l1: 13.4196\tvalid_1's l1: 14.7674\n",
      "[3400]\tvalid_0's l1: 13.4021\tvalid_1's l1: 14.7672\n",
      "[3450]\tvalid_0's l1: 13.3855\tvalid_1's l1: 14.7667\n",
      "[3500]\tvalid_0's l1: 13.3681\tvalid_1's l1: 14.7658\n",
      "[3550]\tvalid_0's l1: 13.351\tvalid_1's l1: 14.7648\n",
      "[3600]\tvalid_0's l1: 13.3337\tvalid_1's l1: 14.7645\n",
      "[3650]\tvalid_0's l1: 13.3165\tvalid_1's l1: 14.764\n",
      "[3700]\tvalid_0's l1: 13.2998\tvalid_1's l1: 14.7645\n",
      "[3750]\tvalid_0's l1: 13.2835\tvalid_1's l1: 14.7645\n",
      "[3800]\tvalid_0's l1: 13.2674\tvalid_1's l1: 14.7648\n",
      "[3850]\tvalid_0's l1: 13.2512\tvalid_1's l1: 14.7653\n",
      "Early stopping, best iteration is:\n",
      "[3647]\tvalid_0's l1: 13.3175\tvalid_1's l1: 14.7638\n",
      "****************************************************************************************************\n",
      "MAE Model 0.06356345258514168\n",
      "MSE Model 0.06343663917051098\n",
      "Merge Model12 0.06359245145019841\n",
      "Training until validation scores don't improve for 250 rounds.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\tvalid_0's l1: 27.6975\tvalid_1's l1: 27.7059\n",
      "[100]\tvalid_0's l1: 23.9586\tvalid_1's l1: 23.9808\n",
      "[150]\tvalid_0's l1: 21.2922\tvalid_1's l1: 21.3326\n",
      "[200]\tvalid_0's l1: 19.4109\tvalid_1's l1: 19.4846\n",
      "[250]\tvalid_0's l1: 18.0956\tvalid_1's l1: 18.2037\n",
      "[300]\tvalid_0's l1: 17.1809\tvalid_1's l1: 17.331\n",
      "[350]\tvalid_0's l1: 16.5384\tvalid_1's l1: 16.7181\n",
      "[400]\tvalid_0's l1: 16.0768\tvalid_1's l1: 16.2856\n",
      "[450]\tvalid_0's l1: 15.7432\tvalid_1's l1: 15.9724\n",
      "[500]\tvalid_0's l1: 15.4977\tvalid_1's l1: 15.7419\n",
      "[550]\tvalid_0's l1: 15.3099\tvalid_1's l1: 15.5686\n",
      "[600]\tvalid_0's l1: 15.159\tvalid_1's l1: 15.4287\n",
      "[650]\tvalid_0's l1: 15.0378\tvalid_1's l1: 15.3206\n",
      "[700]\tvalid_0's l1: 14.9355\tvalid_1's l1: 15.2317\n",
      "[750]\tvalid_0's l1: 14.845\tvalid_1's l1: 15.1584\n",
      "[800]\tvalid_0's l1: 14.7658\tvalid_1's l1: 15.0963\n",
      "[850]\tvalid_0's l1: 14.6951\tvalid_1's l1: 15.0427\n",
      "[900]\tvalid_0's l1: 14.634\tvalid_1's l1: 14.9992\n",
      "[950]\tvalid_0's l1: 14.5805\tvalid_1's l1: 14.9624\n",
      "[1000]\tvalid_0's l1: 14.531\tvalid_1's l1: 14.9303\n",
      "[1050]\tvalid_0's l1: 14.4863\tvalid_1's l1: 14.9048\n",
      "[1100]\tvalid_0's l1: 14.4432\tvalid_1's l1: 14.8803\n",
      "[1150]\tvalid_0's l1: 14.4046\tvalid_1's l1: 14.8602\n",
      "[1200]\tvalid_0's l1: 14.3669\tvalid_1's l1: 14.8416\n",
      "[1250]\tvalid_0's l1: 14.3312\tvalid_1's l1: 14.8255\n",
      "[1300]\tvalid_0's l1: 14.2967\tvalid_1's l1: 14.8104\n",
      "[1350]\tvalid_0's l1: 14.2647\tvalid_1's l1: 14.7969\n",
      "[1400]\tvalid_0's l1: 14.2339\tvalid_1's l1: 14.7845\n",
      "[1450]\tvalid_0's l1: 14.2046\tvalid_1's l1: 14.7742\n",
      "[1500]\tvalid_0's l1: 14.1763\tvalid_1's l1: 14.7645\n",
      "[1550]\tvalid_0's l1: 14.1483\tvalid_1's l1: 14.755\n",
      "[1600]\tvalid_0's l1: 14.1231\tvalid_1's l1: 14.7474\n",
      "[1650]\tvalid_0's l1: 14.0979\tvalid_1's l1: 14.7405\n",
      "[1700]\tvalid_0's l1: 14.0724\tvalid_1's l1: 14.7335\n",
      "[1750]\tvalid_0's l1: 14.0476\tvalid_1's l1: 14.7268\n",
      "[1800]\tvalid_0's l1: 14.0233\tvalid_1's l1: 14.7196\n",
      "[1850]\tvalid_0's l1: 14.0008\tvalid_1's l1: 14.7134\n",
      "[1900]\tvalid_0's l1: 13.9784\tvalid_1's l1: 14.7077\n",
      "[1950]\tvalid_0's l1: 13.9561\tvalid_1's l1: 14.7037\n",
      "[2000]\tvalid_0's l1: 13.934\tvalid_1's l1: 14.6985\n",
      "[2050]\tvalid_0's l1: 13.9123\tvalid_1's l1: 14.6938\n",
      "[2100]\tvalid_0's l1: 13.8915\tvalid_1's l1: 14.6898\n",
      "[2150]\tvalid_0's l1: 13.8707\tvalid_1's l1: 14.6863\n",
      "[2200]\tvalid_0's l1: 13.8501\tvalid_1's l1: 14.6822\n",
      "[2250]\tvalid_0's l1: 13.8298\tvalid_1's l1: 14.6781\n",
      "[2300]\tvalid_0's l1: 13.8089\tvalid_1's l1: 14.6733\n",
      "[2350]\tvalid_0's l1: 13.7888\tvalid_1's l1: 14.6699\n",
      "[2400]\tvalid_0's l1: 13.7692\tvalid_1's l1: 14.6664\n",
      "[2450]\tvalid_0's l1: 13.7491\tvalid_1's l1: 14.6627\n",
      "[2500]\tvalid_0's l1: 13.7302\tvalid_1's l1: 14.6599\n",
      "[2550]\tvalid_0's l1: 13.7109\tvalid_1's l1: 14.6566\n",
      "[2600]\tvalid_0's l1: 13.6926\tvalid_1's l1: 14.6546\n",
      "[2650]\tvalid_0's l1: 13.674\tvalid_1's l1: 14.6516\n",
      "[2700]\tvalid_0's l1: 13.6561\tvalid_1's l1: 14.6489\n",
      "[2750]\tvalid_0's l1: 13.6383\tvalid_1's l1: 14.6461\n",
      "[2800]\tvalid_0's l1: 13.621\tvalid_1's l1: 14.6435\n",
      "[2850]\tvalid_0's l1: 13.6041\tvalid_1's l1: 14.6414\n",
      "[2900]\tvalid_0's l1: 13.5868\tvalid_1's l1: 14.639\n",
      "[2950]\tvalid_0's l1: 13.5702\tvalid_1's l1: 14.6369\n",
      "[3000]\tvalid_0's l1: 13.5534\tvalid_1's l1: 14.6349\n",
      "[3050]\tvalid_0's l1: 13.5369\tvalid_1's l1: 14.6331\n",
      "[3100]\tvalid_0's l1: 13.5201\tvalid_1's l1: 14.6303\n",
      "[3150]\tvalid_0's l1: 13.5036\tvalid_1's l1: 14.6279\n",
      "[3200]\tvalid_0's l1: 13.4872\tvalid_1's l1: 14.6262\n",
      "[3250]\tvalid_0's l1: 13.4704\tvalid_1's l1: 14.6233\n",
      "[3300]\tvalid_0's l1: 13.4548\tvalid_1's l1: 14.6219\n",
      "[3350]\tvalid_0's l1: 13.4397\tvalid_1's l1: 14.6208\n",
      "[3400]\tvalid_0's l1: 13.4244\tvalid_1's l1: 14.6189\n",
      "[3450]\tvalid_0's l1: 13.4093\tvalid_1's l1: 14.6177\n",
      "[3500]\tvalid_0's l1: 13.3943\tvalid_1's l1: 14.6158\n",
      "[3550]\tvalid_0's l1: 13.3788\tvalid_1's l1: 14.6142\n",
      "[3600]\tvalid_0's l1: 13.3644\tvalid_1's l1: 14.6128\n",
      "[3650]\tvalid_0's l1: 13.35\tvalid_1's l1: 14.6116\n",
      "[3700]\tvalid_0's l1: 13.3358\tvalid_1's l1: 14.6103\n",
      "[3750]\tvalid_0's l1: 13.3223\tvalid_1's l1: 14.6093\n",
      "[3800]\tvalid_0's l1: 13.309\tvalid_1's l1: 14.6074\n",
      "[3850]\tvalid_0's l1: 13.2956\tvalid_1's l1: 14.6063\n",
      "[3900]\tvalid_0's l1: 13.282\tvalid_1's l1: 14.6057\n",
      "[3950]\tvalid_0's l1: 13.2678\tvalid_1's l1: 14.605\n",
      "[4000]\tvalid_0's l1: 13.2553\tvalid_1's l1: 14.6045\n",
      "[4050]\tvalid_0's l1: 13.2421\tvalid_1's l1: 14.6041\n",
      "[4100]\tvalid_0's l1: 13.2276\tvalid_1's l1: 14.6034\n",
      "[4150]\tvalid_0's l1: 13.2147\tvalid_1's l1: 14.6029\n",
      "[4200]\tvalid_0's l1: 13.2015\tvalid_1's l1: 14.6025\n",
      "[4250]\tvalid_0's l1: 13.188\tvalid_1's l1: 14.6016\n",
      "[4300]\tvalid_0's l1: 13.1746\tvalid_1's l1: 14.6007\n",
      "[4350]\tvalid_0's l1: 13.1617\tvalid_1's l1: 14.6003\n",
      "[4400]\tvalid_0's l1: 13.1477\tvalid_1's l1: 14.5996\n",
      "[4450]\tvalid_0's l1: 13.1344\tvalid_1's l1: 14.5989\n",
      "[4500]\tvalid_0's l1: 13.1219\tvalid_1's l1: 14.5984\n",
      "[4550]\tvalid_0's l1: 13.1093\tvalid_1's l1: 14.5973\n",
      "[4600]\tvalid_0's l1: 13.097\tvalid_1's l1: 14.5968\n",
      "[4650]\tvalid_0's l1: 13.0849\tvalid_1's l1: 14.5957\n",
      "[4700]\tvalid_0's l1: 13.0723\tvalid_1's l1: 14.5949\n",
      "[4750]\tvalid_0's l1: 13.059\tvalid_1's l1: 14.5942\n",
      "[4800]\tvalid_0's l1: 13.0464\tvalid_1's l1: 14.5934\n",
      "[4850]\tvalid_0's l1: 13.0341\tvalid_1's l1: 14.5928\n",
      "[4900]\tvalid_0's l1: 13.022\tvalid_1's l1: 14.5927\n",
      "[4950]\tvalid_0's l1: 13.0102\tvalid_1's l1: 14.5918\n",
      "[5000]\tvalid_0's l1: 12.9982\tvalid_1's l1: 14.5913\n",
      "[5050]\tvalid_0's l1: 12.9863\tvalid_1's l1: 14.5906\n",
      "[5100]\tvalid_0's l1: 12.9744\tvalid_1's l1: 14.5898\n",
      "[5150]\tvalid_0's l1: 12.9626\tvalid_1's l1: 14.5893\n",
      "[5200]\tvalid_0's l1: 12.9509\tvalid_1's l1: 14.5887\n",
      "[5250]\tvalid_0's l1: 12.9393\tvalid_1's l1: 14.5881\n",
      "[5300]\tvalid_0's l1: 12.9279\tvalid_1's l1: 14.5876\n",
      "[5350]\tvalid_0's l1: 12.9149\tvalid_1's l1: 14.5872\n",
      "[5400]\tvalid_0's l1: 12.9026\tvalid_1's l1: 14.587\n",
      "[5450]\tvalid_0's l1: 12.8908\tvalid_1's l1: 14.587\n",
      "[5500]\tvalid_0's l1: 12.8801\tvalid_1's l1: 14.5865\n",
      "[5550]\tvalid_0's l1: 12.8692\tvalid_1's l1: 14.5861\n",
      "[5600]\tvalid_0's l1: 12.8574\tvalid_1's l1: 14.5857\n",
      "[5650]\tvalid_0's l1: 12.8471\tvalid_1's l1: 14.5847\n",
      "[5700]\tvalid_0's l1: 12.8353\tvalid_1's l1: 14.5842\n",
      "[5750]\tvalid_0's l1: 12.8236\tvalid_1's l1: 14.5836\n",
      "[5800]\tvalid_0's l1: 12.8134\tvalid_1's l1: 14.5836\n",
      "[5850]\tvalid_0's l1: 12.8036\tvalid_1's l1: 14.5826\n",
      "[5900]\tvalid_0's l1: 12.7948\tvalid_1's l1: 14.5819\n",
      "[5950]\tvalid_0's l1: 12.7869\tvalid_1's l1: 14.5819\n",
      "[6000]\tvalid_0's l1: 12.7783\tvalid_1's l1: 14.5814\n",
      "[6050]\tvalid_0's l1: 12.7701\tvalid_1's l1: 14.5815\n",
      "[6100]\tvalid_0's l1: 12.761\tvalid_1's l1: 14.5812\n",
      "[6150]\tvalid_0's l1: 12.7525\tvalid_1's l1: 14.5812\n",
      "[6200]\tvalid_0's l1: 12.7453\tvalid_1's l1: 14.5811\n",
      "[6250]\tvalid_0's l1: 12.7373\tvalid_1's l1: 14.5811\n",
      "[6300]\tvalid_0's l1: 12.7303\tvalid_1's l1: 14.581\n",
      "[6350]\tvalid_0's l1: 12.723\tvalid_1's l1: 14.581\n",
      "[6400]\tvalid_0's l1: 12.7135\tvalid_1's l1: 14.5808\n",
      "[6450]\tvalid_0's l1: 12.7038\tvalid_1's l1: 14.5805\n",
      "[6500]\tvalid_0's l1: 12.6939\tvalid_1's l1: 14.5806\n",
      "[6550]\tvalid_0's l1: 12.685\tvalid_1's l1: 14.5807\n",
      "[6600]\tvalid_0's l1: 12.675\tvalid_1's l1: 14.5806\n",
      "[6650]\tvalid_0's l1: 12.6654\tvalid_1's l1: 14.5803\n",
      "[6700]\tvalid_0's l1: 12.6547\tvalid_1's l1: 14.5798\n",
      "[6750]\tvalid_0's l1: 12.6467\tvalid_1's l1: 14.5798\n",
      "[6800]\tvalid_0's l1: 12.6384\tvalid_1's l1: 14.5792\n",
      "[6850]\tvalid_0's l1: 12.6292\tvalid_1's l1: 14.5786\n",
      "[6900]\tvalid_0's l1: 12.6195\tvalid_1's l1: 14.5785\n",
      "[6950]\tvalid_0's l1: 12.6097\tvalid_1's l1: 14.5781\n",
      "[7000]\tvalid_0's l1: 12.6003\tvalid_1's l1: 14.5781\n",
      "[7050]\tvalid_0's l1: 12.591\tvalid_1's l1: 14.5775\n",
      "[7100]\tvalid_0's l1: 12.5819\tvalid_1's l1: 14.5769\n",
      "[7150]\tvalid_0's l1: 12.5736\tvalid_1's l1: 14.5766\n",
      "[7200]\tvalid_0's l1: 12.5637\tvalid_1's l1: 14.5759\n",
      "[7250]\tvalid_0's l1: 12.5549\tvalid_1's l1: 14.5751\n",
      "[7300]\tvalid_0's l1: 12.5454\tvalid_1's l1: 14.5751\n",
      "[7350]\tvalid_0's l1: 12.5364\tvalid_1's l1: 14.5747\n",
      "[7400]\tvalid_0's l1: 12.5273\tvalid_1's l1: 14.5748\n",
      "[7450]\tvalid_0's l1: 12.5183\tvalid_1's l1: 14.575\n",
      "[7500]\tvalid_0's l1: 12.5092\tvalid_1's l1: 14.5748\n",
      "[7550]\tvalid_0's l1: 12.4993\tvalid_1's l1: 14.5744\n",
      "[7600]\tvalid_0's l1: 12.4901\tvalid_1's l1: 14.5743\n",
      "[7650]\tvalid_0's l1: 12.4812\tvalid_1's l1: 14.5742\n",
      "[7700]\tvalid_0's l1: 12.473\tvalid_1's l1: 14.5746\n",
      "[7750]\tvalid_0's l1: 12.464\tvalid_1's l1: 14.5744\n",
      "[7800]\tvalid_0's l1: 12.4556\tvalid_1's l1: 14.5744\n",
      "[7850]\tvalid_0's l1: 12.4483\tvalid_1's l1: 14.5741\n",
      "[7900]\tvalid_0's l1: 12.4391\tvalid_1's l1: 14.5739\n",
      "[7950]\tvalid_0's l1: 12.4307\tvalid_1's l1: 14.5735\n",
      "[8000]\tvalid_0's l1: 12.4227\tvalid_1's l1: 14.5735\n",
      "[8050]\tvalid_0's l1: 12.4136\tvalid_1's l1: 14.5735\n",
      "[8100]\tvalid_0's l1: 12.4053\tvalid_1's l1: 14.5736\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[8150]\tvalid_0's l1: 12.3976\tvalid_1's l1: 14.5737\n",
      "[8200]\tvalid_0's l1: 12.389\tvalid_1's l1: 14.5736\n",
      "[8250]\tvalid_0's l1: 12.3799\tvalid_1's l1: 14.573\n",
      "[8300]\tvalid_0's l1: 12.3705\tvalid_1's l1: 14.5723\n",
      "[8350]\tvalid_0's l1: 12.3619\tvalid_1's l1: 14.572\n",
      "[8400]\tvalid_0's l1: 12.3534\tvalid_1's l1: 14.5718\n",
      "[8450]\tvalid_0's l1: 12.3452\tvalid_1's l1: 14.572\n",
      "[8500]\tvalid_0's l1: 12.3369\tvalid_1's l1: 14.5717\n",
      "[8550]\tvalid_0's l1: 12.3295\tvalid_1's l1: 14.5717\n",
      "[8600]\tvalid_0's l1: 12.3217\tvalid_1's l1: 14.5715\n",
      "[8650]\tvalid_0's l1: 12.3137\tvalid_1's l1: 14.5715\n",
      "[8700]\tvalid_0's l1: 12.3059\tvalid_1's l1: 14.5715\n",
      "[8750]\tvalid_0's l1: 12.2982\tvalid_1's l1: 14.5715\n",
      "[8800]\tvalid_0's l1: 12.2909\tvalid_1's l1: 14.5713\n",
      "[8850]\tvalid_0's l1: 12.2833\tvalid_1's l1: 14.5713\n",
      "[8900]\tvalid_0's l1: 12.2781\tvalid_1's l1: 14.5715\n",
      "[8950]\tvalid_0's l1: 12.2725\tvalid_1's l1: 14.5717\n",
      "[9000]\tvalid_0's l1: 12.2659\tvalid_1's l1: 14.5718\n",
      "[9050]\tvalid_0's l1: 12.2597\tvalid_1's l1: 14.572\n",
      "Early stopping, best iteration is:\n",
      "[8845]\tvalid_0's l1: 12.2839\tvalid_1's l1: 14.5711\n",
      "Training until validation scores don't improve for 250 rounds.\n",
      "[50]\tvalid_0's l1: 28.1664\tvalid_1's l1: 28.3146\n",
      "[100]\tvalid_0's l1: 24.1824\tvalid_1's l1: 24.3413\n",
      "[150]\tvalid_0's l1: 21.3556\tvalid_1's l1: 21.5063\n",
      "[200]\tvalid_0's l1: 19.3871\tvalid_1's l1: 19.5378\n",
      "[250]\tvalid_0's l1: 18.0325\tvalid_1's l1: 18.1977\n",
      "[300]\tvalid_0's l1: 17.1122\tvalid_1's l1: 17.2891\n",
      "[350]\tvalid_0's l1: 16.4833\tvalid_1's l1: 16.6676\n",
      "[400]\tvalid_0's l1: 16.0429\tvalid_1's l1: 16.2323\n",
      "[450]\tvalid_0's l1: 15.7288\tvalid_1's l1: 15.9266\n",
      "[500]\tvalid_0's l1: 15.4976\tvalid_1's l1: 15.7006\n",
      "[550]\tvalid_0's l1: 15.3232\tvalid_1's l1: 15.5339\n",
      "[600]\tvalid_0's l1: 15.1828\tvalid_1's l1: 15.3978\n",
      "[650]\tvalid_0's l1: 15.0706\tvalid_1's l1: 15.291\n",
      "[700]\tvalid_0's l1: 14.9751\tvalid_1's l1: 15.2028\n",
      "[750]\tvalid_0's l1: 14.8912\tvalid_1's l1: 15.1291\n",
      "[800]\tvalid_0's l1: 14.8164\tvalid_1's l1: 15.0677\n",
      "[850]\tvalid_0's l1: 14.7503\tvalid_1's l1: 15.0151\n",
      "[900]\tvalid_0's l1: 14.6941\tvalid_1's l1: 14.9719\n",
      "[950]\tvalid_0's l1: 14.6414\tvalid_1's l1: 14.9357\n",
      "[1000]\tvalid_0's l1: 14.5948\tvalid_1's l1: 14.9053\n",
      "[1050]\tvalid_0's l1: 14.5495\tvalid_1's l1: 14.8757\n",
      "[1100]\tvalid_0's l1: 14.5066\tvalid_1's l1: 14.8494\n",
      "[1150]\tvalid_0's l1: 14.4659\tvalid_1's l1: 14.8254\n",
      "[1200]\tvalid_0's l1: 14.4265\tvalid_1's l1: 14.8042\n",
      "[1250]\tvalid_0's l1: 14.3907\tvalid_1's l1: 14.7856\n",
      "[1300]\tvalid_0's l1: 14.3556\tvalid_1's l1: 14.7684\n",
      "[1350]\tvalid_0's l1: 14.3235\tvalid_1's l1: 14.7548\n",
      "[1400]\tvalid_0's l1: 14.2921\tvalid_1's l1: 14.7413\n",
      "[1450]\tvalid_0's l1: 14.2637\tvalid_1's l1: 14.7304\n",
      "[1500]\tvalid_0's l1: 14.2353\tvalid_1's l1: 14.7194\n",
      "[1550]\tvalid_0's l1: 14.2082\tvalid_1's l1: 14.7096\n",
      "[1600]\tvalid_0's l1: 14.1818\tvalid_1's l1: 14.7008\n",
      "[1650]\tvalid_0's l1: 14.1558\tvalid_1's l1: 14.6935\n",
      "[1700]\tvalid_0's l1: 14.1303\tvalid_1's l1: 14.6855\n",
      "[1750]\tvalid_0's l1: 14.1063\tvalid_1's l1: 14.6792\n",
      "[1800]\tvalid_0's l1: 14.0819\tvalid_1's l1: 14.6731\n",
      "[1850]\tvalid_0's l1: 14.0586\tvalid_1's l1: 14.668\n",
      "[1900]\tvalid_0's l1: 14.0361\tvalid_1's l1: 14.6633\n",
      "[1950]\tvalid_0's l1: 14.0131\tvalid_1's l1: 14.6577\n",
      "[2000]\tvalid_0's l1: 13.9914\tvalid_1's l1: 14.6524\n",
      "[2050]\tvalid_0's l1: 13.9693\tvalid_1's l1: 14.6469\n",
      "[2100]\tvalid_0's l1: 13.9478\tvalid_1's l1: 14.6432\n",
      "[2150]\tvalid_0's l1: 13.9255\tvalid_1's l1: 14.6386\n",
      "[2200]\tvalid_0's l1: 13.9048\tvalid_1's l1: 14.6341\n",
      "[2250]\tvalid_0's l1: 13.884\tvalid_1's l1: 14.6304\n",
      "[2300]\tvalid_0's l1: 13.8636\tvalid_1's l1: 14.627\n",
      "[2350]\tvalid_0's l1: 13.8431\tvalid_1's l1: 14.6235\n",
      "[2400]\tvalid_0's l1: 13.8227\tvalid_1's l1: 14.6186\n",
      "[2450]\tvalid_0's l1: 13.8035\tvalid_1's l1: 14.6157\n",
      "[2500]\tvalid_0's l1: 13.784\tvalid_1's l1: 14.6127\n",
      "[2550]\tvalid_0's l1: 13.7654\tvalid_1's l1: 14.6099\n",
      "[2600]\tvalid_0's l1: 13.7464\tvalid_1's l1: 14.6069\n",
      "[2650]\tvalid_0's l1: 13.7275\tvalid_1's l1: 14.6048\n",
      "[2700]\tvalid_0's l1: 13.7089\tvalid_1's l1: 14.6022\n",
      "[2750]\tvalid_0's l1: 13.6908\tvalid_1's l1: 14.6006\n",
      "[2800]\tvalid_0's l1: 13.6716\tvalid_1's l1: 14.5989\n",
      "[2850]\tvalid_0's l1: 13.6532\tvalid_1's l1: 14.5966\n",
      "[2900]\tvalid_0's l1: 13.6352\tvalid_1's l1: 14.5948\n",
      "[2950]\tvalid_0's l1: 13.6169\tvalid_1's l1: 14.5929\n",
      "[3000]\tvalid_0's l1: 13.5991\tvalid_1's l1: 14.5903\n",
      "[3050]\tvalid_0's l1: 13.5813\tvalid_1's l1: 14.5893\n",
      "[3100]\tvalid_0's l1: 13.5641\tvalid_1's l1: 14.5878\n",
      "[3150]\tvalid_0's l1: 13.5467\tvalid_1's l1: 14.5863\n",
      "[3200]\tvalid_0's l1: 13.5296\tvalid_1's l1: 14.5849\n",
      "[3250]\tvalid_0's l1: 13.5122\tvalid_1's l1: 14.5838\n",
      "[3300]\tvalid_0's l1: 13.4947\tvalid_1's l1: 14.5821\n",
      "[3350]\tvalid_0's l1: 13.4774\tvalid_1's l1: 14.5808\n",
      "[3400]\tvalid_0's l1: 13.4604\tvalid_1's l1: 14.5788\n",
      "[3450]\tvalid_0's l1: 13.4428\tvalid_1's l1: 14.5772\n",
      "[3500]\tvalid_0's l1: 13.4256\tvalid_1's l1: 14.5757\n",
      "[3550]\tvalid_0's l1: 13.4087\tvalid_1's l1: 14.5745\n",
      "[3600]\tvalid_0's l1: 13.392\tvalid_1's l1: 14.5734\n",
      "[3650]\tvalid_0's l1: 13.3757\tvalid_1's l1: 14.5721\n",
      "[3700]\tvalid_0's l1: 13.36\tvalid_1's l1: 14.5714\n",
      "[3750]\tvalid_0's l1: 13.344\tvalid_1's l1: 14.5712\n",
      "[3800]\tvalid_0's l1: 13.3282\tvalid_1's l1: 14.5708\n",
      "[3850]\tvalid_0's l1: 13.3119\tvalid_1's l1: 14.5696\n",
      "[3900]\tvalid_0's l1: 13.2956\tvalid_1's l1: 14.5688\n",
      "[3950]\tvalid_0's l1: 13.2794\tvalid_1's l1: 14.5683\n",
      "[4000]\tvalid_0's l1: 13.2634\tvalid_1's l1: 14.568\n",
      "[4050]\tvalid_0's l1: 13.2475\tvalid_1's l1: 14.5666\n",
      "[4100]\tvalid_0's l1: 13.2309\tvalid_1's l1: 14.566\n",
      "[4150]\tvalid_0's l1: 13.2147\tvalid_1's l1: 14.5661\n",
      "[4200]\tvalid_0's l1: 13.1993\tvalid_1's l1: 14.5655\n",
      "[4250]\tvalid_0's l1: 13.1836\tvalid_1's l1: 14.5649\n",
      "[4300]\tvalid_0's l1: 13.168\tvalid_1's l1: 14.5645\n",
      "[4350]\tvalid_0's l1: 13.1527\tvalid_1's l1: 14.5632\n",
      "[4400]\tvalid_0's l1: 13.1376\tvalid_1's l1: 14.5631\n",
      "[4450]\tvalid_0's l1: 13.1224\tvalid_1's l1: 14.5621\n",
      "[4500]\tvalid_0's l1: 13.1072\tvalid_1's l1: 14.5613\n",
      "[4550]\tvalid_0's l1: 13.0916\tvalid_1's l1: 14.5611\n",
      "[4600]\tvalid_0's l1: 13.0757\tvalid_1's l1: 14.5603\n",
      "[4650]\tvalid_0's l1: 13.0601\tvalid_1's l1: 14.5589\n",
      "[4700]\tvalid_0's l1: 13.0445\tvalid_1's l1: 14.5589\n",
      "[4750]\tvalid_0's l1: 13.029\tvalid_1's l1: 14.5586\n",
      "[4800]\tvalid_0's l1: 13.014\tvalid_1's l1: 14.5579\n",
      "[4850]\tvalid_0's l1: 12.9985\tvalid_1's l1: 14.5564\n",
      "[4900]\tvalid_0's l1: 12.9836\tvalid_1's l1: 14.5555\n",
      "[4950]\tvalid_0's l1: 12.9686\tvalid_1's l1: 14.5551\n",
      "[5000]\tvalid_0's l1: 12.9535\tvalid_1's l1: 14.5548\n",
      "[5050]\tvalid_0's l1: 12.9389\tvalid_1's l1: 14.5538\n",
      "[5100]\tvalid_0's l1: 12.9244\tvalid_1's l1: 14.5534\n",
      "[5150]\tvalid_0's l1: 12.9101\tvalid_1's l1: 14.553\n",
      "[5200]\tvalid_0's l1: 12.8954\tvalid_1's l1: 14.553\n",
      "[5250]\tvalid_0's l1: 12.8814\tvalid_1's l1: 14.5524\n",
      "[5300]\tvalid_0's l1: 12.8667\tvalid_1's l1: 14.5518\n",
      "[5350]\tvalid_0's l1: 12.8518\tvalid_1's l1: 14.5516\n",
      "[5400]\tvalid_0's l1: 12.8374\tvalid_1's l1: 14.5521\n",
      "[5450]\tvalid_0's l1: 12.8228\tvalid_1's l1: 14.552\n",
      "[5500]\tvalid_0's l1: 12.8089\tvalid_1's l1: 14.5522\n",
      "[5550]\tvalid_0's l1: 12.7944\tvalid_1's l1: 14.5518\n",
      "[5600]\tvalid_0's l1: 12.7797\tvalid_1's l1: 14.5509\n",
      "[5650]\tvalid_0's l1: 12.7651\tvalid_1's l1: 14.5508\n",
      "[5700]\tvalid_0's l1: 12.7507\tvalid_1's l1: 14.5511\n",
      "[5750]\tvalid_0's l1: 12.7365\tvalid_1's l1: 14.5511\n",
      "[5800]\tvalid_0's l1: 12.7224\tvalid_1's l1: 14.5505\n",
      "[5850]\tvalid_0's l1: 12.708\tvalid_1's l1: 14.5505\n",
      "[5900]\tvalid_0's l1: 12.6936\tvalid_1's l1: 14.5502\n",
      "[5950]\tvalid_0's l1: 12.6792\tvalid_1's l1: 14.5501\n",
      "[6000]\tvalid_0's l1: 12.6658\tvalid_1's l1: 14.5504\n",
      "[6050]\tvalid_0's l1: 12.6514\tvalid_1's l1: 14.5495\n",
      "[6100]\tvalid_0's l1: 12.637\tvalid_1's l1: 14.5492\n",
      "[6150]\tvalid_0's l1: 12.6227\tvalid_1's l1: 14.5489\n",
      "[6200]\tvalid_0's l1: 12.6088\tvalid_1's l1: 14.5491\n",
      "[6250]\tvalid_0's l1: 12.5951\tvalid_1's l1: 14.5489\n",
      "[6300]\tvalid_0's l1: 12.5808\tvalid_1's l1: 14.548\n",
      "[6350]\tvalid_0's l1: 12.5666\tvalid_1's l1: 14.5478\n",
      "[6400]\tvalid_0's l1: 12.5529\tvalid_1's l1: 14.5476\n",
      "[6450]\tvalid_0's l1: 12.539\tvalid_1's l1: 14.5479\n",
      "[6500]\tvalid_0's l1: 12.525\tvalid_1's l1: 14.5479\n",
      "[6550]\tvalid_0's l1: 12.5116\tvalid_1's l1: 14.5481\n",
      "[6600]\tvalid_0's l1: 12.4976\tvalid_1's l1: 14.548\n",
      "[6650]\tvalid_0's l1: 12.4842\tvalid_1's l1: 14.5483\n",
      "Early stopping, best iteration is:\n",
      "[6410]\tvalid_0's l1: 12.5501\tvalid_1's l1: 14.5473\n",
      "****************************************************************************************************\n",
      "MAE Model 0.06422149465677383\n",
      "MSE Model 0.06431968194428975\n",
      "Merge Model12 0.06436835293198238\n",
      "Training until validation scores don't improve for 250 rounds.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\tvalid_0's l1: 27.7947\tvalid_1's l1: 27.2948\n",
      "[100]\tvalid_0's l1: 24.0329\tvalid_1's l1: 23.6702\n",
      "[150]\tvalid_0's l1: 21.3545\tvalid_1's l1: 21.0899\n",
      "[200]\tvalid_0's l1: 19.4604\tvalid_1's l1: 19.2782\n",
      "[250]\tvalid_0's l1: 18.1247\tvalid_1's l1: 18.0238\n",
      "[300]\tvalid_0's l1: 17.1965\tvalid_1's l1: 17.1578\n",
      "[350]\tvalid_0's l1: 16.541\tvalid_1's l1: 16.5618\n",
      "[400]\tvalid_0's l1: 16.0744\tvalid_1's l1: 16.1419\n",
      "[450]\tvalid_0's l1: 15.7358\tvalid_1's l1: 15.8459\n",
      "[500]\tvalid_0's l1: 15.4887\tvalid_1's l1: 15.6298\n",
      "[550]\tvalid_0's l1: 15.2981\tvalid_1's l1: 15.4734\n",
      "[600]\tvalid_0's l1: 15.1451\tvalid_1's l1: 15.3541\n",
      "[650]\tvalid_0's l1: 15.021\tvalid_1's l1: 15.2612\n",
      "[700]\tvalid_0's l1: 14.9177\tvalid_1's l1: 15.1886\n",
      "[750]\tvalid_0's l1: 14.8263\tvalid_1's l1: 15.1278\n",
      "[800]\tvalid_0's l1: 14.7476\tvalid_1's l1: 15.0792\n",
      "[850]\tvalid_0's l1: 14.6788\tvalid_1's l1: 15.0401\n",
      "[900]\tvalid_0's l1: 14.6158\tvalid_1's l1: 15.0067\n",
      "[950]\tvalid_0's l1: 14.5587\tvalid_1's l1: 14.9763\n",
      "[1000]\tvalid_0's l1: 14.5067\tvalid_1's l1: 14.951\n",
      "[1050]\tvalid_0's l1: 14.4575\tvalid_1's l1: 14.927\n",
      "[1100]\tvalid_0's l1: 14.4134\tvalid_1's l1: 14.9086\n",
      "[1150]\tvalid_0's l1: 14.3717\tvalid_1's l1: 14.8924\n",
      "[1200]\tvalid_0's l1: 14.3346\tvalid_1's l1: 14.8793\n",
      "[1250]\tvalid_0's l1: 14.2998\tvalid_1's l1: 14.8681\n",
      "[1300]\tvalid_0's l1: 14.2658\tvalid_1's l1: 14.8573\n",
      "[1350]\tvalid_0's l1: 14.2333\tvalid_1's l1: 14.8468\n",
      "[1400]\tvalid_0's l1: 14.2024\tvalid_1's l1: 14.837\n",
      "[1450]\tvalid_0's l1: 14.1736\tvalid_1's l1: 14.8306\n",
      "[1500]\tvalid_0's l1: 14.1449\tvalid_1's l1: 14.8231\n",
      "[1550]\tvalid_0's l1: 14.1164\tvalid_1's l1: 14.8154\n",
      "[1600]\tvalid_0's l1: 14.0895\tvalid_1's l1: 14.8089\n",
      "[1650]\tvalid_0's l1: 14.0645\tvalid_1's l1: 14.8053\n",
      "[1700]\tvalid_0's l1: 14.0386\tvalid_1's l1: 14.8007\n",
      "[1750]\tvalid_0's l1: 14.0142\tvalid_1's l1: 14.798\n",
      "[1800]\tvalid_0's l1: 13.9907\tvalid_1's l1: 14.7936\n",
      "[1850]\tvalid_0's l1: 13.9669\tvalid_1's l1: 14.7892\n",
      "[1900]\tvalid_0's l1: 13.9435\tvalid_1's l1: 14.7855\n",
      "[1950]\tvalid_0's l1: 13.9212\tvalid_1's l1: 14.7822\n",
      "[2000]\tvalid_0's l1: 13.8984\tvalid_1's l1: 14.7789\n",
      "[2050]\tvalid_0's l1: 13.8756\tvalid_1's l1: 14.7748\n",
      "[2100]\tvalid_0's l1: 13.8532\tvalid_1's l1: 14.771\n",
      "[2150]\tvalid_0's l1: 13.8323\tvalid_1's l1: 14.769\n",
      "[2200]\tvalid_0's l1: 13.8116\tvalid_1's l1: 14.7663\n",
      "[2250]\tvalid_0's l1: 13.7914\tvalid_1's l1: 14.7643\n",
      "[2300]\tvalid_0's l1: 13.7714\tvalid_1's l1: 14.762\n",
      "[2350]\tvalid_0's l1: 13.7518\tvalid_1's l1: 14.7602\n",
      "[2400]\tvalid_0's l1: 13.7317\tvalid_1's l1: 14.758\n",
      "[2450]\tvalid_0's l1: 13.7118\tvalid_1's l1: 14.7558\n",
      "[2500]\tvalid_0's l1: 13.6929\tvalid_1's l1: 14.7543\n",
      "[2550]\tvalid_0's l1: 13.673\tvalid_1's l1: 14.7526\n",
      "[2600]\tvalid_0's l1: 13.6541\tvalid_1's l1: 14.7506\n",
      "[2650]\tvalid_0's l1: 13.6354\tvalid_1's l1: 14.7491\n",
      "[2700]\tvalid_0's l1: 13.6165\tvalid_1's l1: 14.7479\n",
      "[2750]\tvalid_0's l1: 13.5981\tvalid_1's l1: 14.7459\n",
      "[2800]\tvalid_0's l1: 13.5812\tvalid_1's l1: 14.745\n",
      "[2850]\tvalid_0's l1: 13.5636\tvalid_1's l1: 14.7436\n",
      "[2900]\tvalid_0's l1: 13.5452\tvalid_1's l1: 14.7418\n",
      "[2950]\tvalid_0's l1: 13.5286\tvalid_1's l1: 14.7414\n",
      "[3000]\tvalid_0's l1: 13.5119\tvalid_1's l1: 14.7404\n",
      "[3050]\tvalid_0's l1: 13.4958\tvalid_1's l1: 14.7403\n",
      "[3100]\tvalid_0's l1: 13.4788\tvalid_1's l1: 14.7388\n",
      "[3150]\tvalid_0's l1: 13.4618\tvalid_1's l1: 14.7376\n",
      "[3200]\tvalid_0's l1: 13.4453\tvalid_1's l1: 14.7367\n",
      "[3250]\tvalid_0's l1: 13.4292\tvalid_1's l1: 14.736\n",
      "[3300]\tvalid_0's l1: 13.4128\tvalid_1's l1: 14.735\n",
      "[3350]\tvalid_0's l1: 13.3969\tvalid_1's l1: 14.7341\n",
      "[3400]\tvalid_0's l1: 13.3808\tvalid_1's l1: 14.7328\n",
      "[3450]\tvalid_0's l1: 13.365\tvalid_1's l1: 14.7325\n",
      "[3500]\tvalid_0's l1: 13.3495\tvalid_1's l1: 14.7319\n",
      "[3550]\tvalid_0's l1: 13.3341\tvalid_1's l1: 14.7319\n",
      "[3600]\tvalid_0's l1: 13.3195\tvalid_1's l1: 14.7313\n",
      "[3650]\tvalid_0's l1: 13.3054\tvalid_1's l1: 14.731\n",
      "[3700]\tvalid_0's l1: 13.2906\tvalid_1's l1: 14.7296\n",
      "[3750]\tvalid_0's l1: 13.2762\tvalid_1's l1: 14.7299\n",
      "[3800]\tvalid_0's l1: 13.2633\tvalid_1's l1: 14.7295\n",
      "[3850]\tvalid_0's l1: 13.2503\tvalid_1's l1: 14.729\n",
      "[3900]\tvalid_0's l1: 13.2352\tvalid_1's l1: 14.7284\n",
      "[3950]\tvalid_0's l1: 13.2211\tvalid_1's l1: 14.7281\n",
      "[4000]\tvalid_0's l1: 13.2075\tvalid_1's l1: 14.7273\n",
      "[4050]\tvalid_0's l1: 13.1933\tvalid_1's l1: 14.7265\n",
      "[4100]\tvalid_0's l1: 13.1785\tvalid_1's l1: 14.7254\n",
      "[4150]\tvalid_0's l1: 13.1665\tvalid_1's l1: 14.7248\n",
      "[4200]\tvalid_0's l1: 13.1533\tvalid_1's l1: 14.7246\n",
      "[4250]\tvalid_0's l1: 13.1395\tvalid_1's l1: 14.7241\n",
      "[4300]\tvalid_0's l1: 13.126\tvalid_1's l1: 14.7234\n",
      "[4350]\tvalid_0's l1: 13.1127\tvalid_1's l1: 14.7228\n",
      "[4400]\tvalid_0's l1: 13.1003\tvalid_1's l1: 14.7222\n",
      "[4450]\tvalid_0's l1: 13.0868\tvalid_1's l1: 14.7218\n",
      "[4500]\tvalid_0's l1: 13.0735\tvalid_1's l1: 14.7214\n",
      "[4550]\tvalid_0's l1: 13.0604\tvalid_1's l1: 14.7205\n",
      "[4600]\tvalid_0's l1: 13.0482\tvalid_1's l1: 14.7206\n",
      "[4650]\tvalid_0's l1: 13.0364\tvalid_1's l1: 14.7198\n",
      "[4700]\tvalid_0's l1: 13.0241\tvalid_1's l1: 14.7199\n",
      "[4750]\tvalid_0's l1: 13.0114\tvalid_1's l1: 14.7193\n",
      "[4800]\tvalid_0's l1: 12.9995\tvalid_1's l1: 14.7194\n",
      "[4850]\tvalid_0's l1: 12.9866\tvalid_1's l1: 14.718\n",
      "[4900]\tvalid_0's l1: 12.9739\tvalid_1's l1: 14.7176\n",
      "[4950]\tvalid_0's l1: 12.9614\tvalid_1's l1: 14.7171\n",
      "[5000]\tvalid_0's l1: 12.9488\tvalid_1's l1: 14.7167\n",
      "[5050]\tvalid_0's l1: 12.9355\tvalid_1's l1: 14.7169\n",
      "[5100]\tvalid_0's l1: 12.9232\tvalid_1's l1: 14.7167\n",
      "[5150]\tvalid_0's l1: 12.912\tvalid_1's l1: 14.7167\n",
      "[5200]\tvalid_0's l1: 12.9011\tvalid_1's l1: 14.7164\n",
      "[5250]\tvalid_0's l1: 12.8889\tvalid_1's l1: 14.7161\n",
      "[5300]\tvalid_0's l1: 12.8773\tvalid_1's l1: 14.7159\n",
      "[5350]\tvalid_0's l1: 12.8667\tvalid_1's l1: 14.7156\n",
      "[5400]\tvalid_0's l1: 12.8552\tvalid_1's l1: 14.7156\n",
      "[5450]\tvalid_0's l1: 12.8444\tvalid_1's l1: 14.7153\n",
      "[5500]\tvalid_0's l1: 12.8334\tvalid_1's l1: 14.7144\n",
      "[5550]\tvalid_0's l1: 12.8235\tvalid_1's l1: 14.7141\n",
      "[5600]\tvalid_0's l1: 12.8132\tvalid_1's l1: 14.7134\n",
      "[5650]\tvalid_0's l1: 12.8036\tvalid_1's l1: 14.7134\n",
      "[5700]\tvalid_0's l1: 12.7953\tvalid_1's l1: 14.7135\n",
      "[5750]\tvalid_0's l1: 12.7852\tvalid_1's l1: 14.7137\n",
      "[5800]\tvalid_0's l1: 12.7742\tvalid_1's l1: 14.7132\n",
      "[5850]\tvalid_0's l1: 12.7639\tvalid_1's l1: 14.7126\n",
      "[5900]\tvalid_0's l1: 12.7529\tvalid_1's l1: 14.712\n",
      "[5950]\tvalid_0's l1: 12.7422\tvalid_1's l1: 14.7118\n",
      "[6000]\tvalid_0's l1: 12.7333\tvalid_1's l1: 14.7119\n",
      "[6050]\tvalid_0's l1: 12.7234\tvalid_1's l1: 14.7117\n",
      "[6100]\tvalid_0's l1: 12.7133\tvalid_1's l1: 14.7117\n",
      "[6150]\tvalid_0's l1: 12.703\tvalid_1's l1: 14.7117\n",
      "[6200]\tvalid_0's l1: 12.6942\tvalid_1's l1: 14.7114\n",
      "[6250]\tvalid_0's l1: 12.687\tvalid_1's l1: 14.7117\n",
      "[6300]\tvalid_0's l1: 12.6805\tvalid_1's l1: 14.7117\n",
      "[6350]\tvalid_0's l1: 12.6743\tvalid_1's l1: 14.7116\n",
      "[6400]\tvalid_0's l1: 12.6652\tvalid_1's l1: 14.7114\n",
      "[6450]\tvalid_0's l1: 12.6564\tvalid_1's l1: 14.7115\n",
      "[6500]\tvalid_0's l1: 12.6482\tvalid_1's l1: 14.7115\n",
      "[6550]\tvalid_0's l1: 12.6385\tvalid_1's l1: 14.7111\n",
      "[6600]\tvalid_0's l1: 12.6287\tvalid_1's l1: 14.711\n",
      "[6650]\tvalid_0's l1: 12.6187\tvalid_1's l1: 14.7104\n",
      "[6700]\tvalid_0's l1: 12.6092\tvalid_1's l1: 14.7104\n",
      "[6750]\tvalid_0's l1: 12.5997\tvalid_1's l1: 14.7101\n",
      "[6800]\tvalid_0's l1: 12.5905\tvalid_1's l1: 14.7098\n",
      "[6850]\tvalid_0's l1: 12.5803\tvalid_1's l1: 14.7098\n",
      "[6900]\tvalid_0's l1: 12.571\tvalid_1's l1: 14.7099\n",
      "[6950]\tvalid_0's l1: 12.5634\tvalid_1's l1: 14.7096\n",
      "[7000]\tvalid_0's l1: 12.5537\tvalid_1's l1: 14.7094\n",
      "[7050]\tvalid_0's l1: 12.5457\tvalid_1's l1: 14.7094\n",
      "[7100]\tvalid_0's l1: 12.5367\tvalid_1's l1: 14.7089\n",
      "[7150]\tvalid_0's l1: 12.529\tvalid_1's l1: 14.7086\n",
      "[7200]\tvalid_0's l1: 12.522\tvalid_1's l1: 14.7087\n",
      "[7250]\tvalid_0's l1: 12.5165\tvalid_1's l1: 14.7085\n",
      "[7300]\tvalid_0's l1: 12.5113\tvalid_1's l1: 14.7085\n",
      "[7350]\tvalid_0's l1: 12.5065\tvalid_1's l1: 14.7084\n",
      "[7400]\tvalid_0's l1: 12.4994\tvalid_1's l1: 14.708\n",
      "[7450]\tvalid_0's l1: 12.4922\tvalid_1's l1: 14.7075\n",
      "[7500]\tvalid_0's l1: 12.4829\tvalid_1's l1: 14.7075\n",
      "[7550]\tvalid_0's l1: 12.474\tvalid_1's l1: 14.7074\n",
      "[7600]\tvalid_0's l1: 12.4659\tvalid_1's l1: 14.7072\n",
      "[7650]\tvalid_0's l1: 12.4585\tvalid_1's l1: 14.7067\n",
      "[7700]\tvalid_0's l1: 12.4512\tvalid_1's l1: 14.7064\n",
      "[7750]\tvalid_0's l1: 12.4421\tvalid_1's l1: 14.7061\n",
      "[7800]\tvalid_0's l1: 12.4332\tvalid_1's l1: 14.7057\n",
      "[7850]\tvalid_0's l1: 12.4247\tvalid_1's l1: 14.7054\n",
      "[7900]\tvalid_0's l1: 12.4165\tvalid_1's l1: 14.7055\n",
      "[7950]\tvalid_0's l1: 12.4074\tvalid_1's l1: 14.7053\n",
      "[8000]\tvalid_0's l1: 12.3989\tvalid_1's l1: 14.7059\n",
      "[8050]\tvalid_0's l1: 12.3905\tvalid_1's l1: 14.7057\n",
      "[8100]\tvalid_0's l1: 12.3819\tvalid_1's l1: 14.7057\n",
      "Early stopping, best iteration is:\n",
      "[7878]\tvalid_0's l1: 12.4201\tvalid_1's l1: 14.7051\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training until validation scores don't improve for 250 rounds.\n",
      "[50]\tvalid_0's l1: 28.2731\tvalid_1's l1: 27.8112\n",
      "[100]\tvalid_0's l1: 24.2584\tvalid_1's l1: 23.9191\n",
      "[150]\tvalid_0's l1: 21.3975\tvalid_1's l1: 21.1752\n",
      "[200]\tvalid_0's l1: 19.3991\tvalid_1's l1: 19.2712\n",
      "[250]\tvalid_0's l1: 18.0264\tvalid_1's l1: 17.973\n",
      "[300]\tvalid_0's l1: 17.0909\tvalid_1's l1: 17.1071\n",
      "[350]\tvalid_0's l1: 16.4526\tvalid_1's l1: 16.5151\n",
      "[400]\tvalid_0's l1: 16.0062\tvalid_1's l1: 16.104\n",
      "[450]\tvalid_0's l1: 15.6889\tvalid_1's l1: 15.8203\n",
      "[500]\tvalid_0's l1: 15.4541\tvalid_1's l1: 15.6212\n",
      "[550]\tvalid_0's l1: 15.2765\tvalid_1's l1: 15.4802\n",
      "[600]\tvalid_0's l1: 15.1318\tvalid_1's l1: 15.3719\n",
      "[650]\tvalid_0's l1: 15.0164\tvalid_1's l1: 15.2915\n",
      "[700]\tvalid_0's l1: 14.9179\tvalid_1's l1: 15.2238\n",
      "[750]\tvalid_0's l1: 14.8322\tvalid_1's l1: 15.1673\n",
      "[800]\tvalid_0's l1: 14.759\tvalid_1's l1: 15.1213\n",
      "[850]\tvalid_0's l1: 14.6942\tvalid_1's l1: 15.0839\n",
      "[900]\tvalid_0's l1: 14.6367\tvalid_1's l1: 15.0547\n",
      "[950]\tvalid_0's l1: 14.5839\tvalid_1's l1: 15.0269\n",
      "[1000]\tvalid_0's l1: 14.5363\tvalid_1's l1: 15.0055\n",
      "[1050]\tvalid_0's l1: 14.4903\tvalid_1's l1: 14.9831\n",
      "[1100]\tvalid_0's l1: 14.4471\tvalid_1's l1: 14.9624\n",
      "[1150]\tvalid_0's l1: 14.4067\tvalid_1's l1: 14.9455\n",
      "[1200]\tvalid_0's l1: 14.3664\tvalid_1's l1: 14.9274\n",
      "[1250]\tvalid_0's l1: 14.3311\tvalid_1's l1: 14.9156\n",
      "[1300]\tvalid_0's l1: 14.2958\tvalid_1's l1: 14.903\n",
      "[1350]\tvalid_0's l1: 14.2637\tvalid_1's l1: 14.8931\n",
      "[1400]\tvalid_0's l1: 14.2331\tvalid_1's l1: 14.8848\n",
      "[1450]\tvalid_0's l1: 14.2043\tvalid_1's l1: 14.8763\n",
      "[1500]\tvalid_0's l1: 14.1755\tvalid_1's l1: 14.8681\n",
      "[1550]\tvalid_0's l1: 14.1484\tvalid_1's l1: 14.8619\n",
      "[1600]\tvalid_0's l1: 14.1221\tvalid_1's l1: 14.8576\n",
      "[1650]\tvalid_0's l1: 14.0968\tvalid_1's l1: 14.8528\n",
      "[1700]\tvalid_0's l1: 14.0711\tvalid_1's l1: 14.8475\n",
      "[1750]\tvalid_0's l1: 14.0472\tvalid_1's l1: 14.8424\n",
      "[1800]\tvalid_0's l1: 14.0232\tvalid_1's l1: 14.837\n",
      "[1850]\tvalid_0's l1: 14.0004\tvalid_1's l1: 14.8342\n",
      "[1900]\tvalid_0's l1: 13.977\tvalid_1's l1: 14.8295\n",
      "[1950]\tvalid_0's l1: 13.9546\tvalid_1's l1: 14.8255\n",
      "[2000]\tvalid_0's l1: 13.9323\tvalid_1's l1: 14.8226\n",
      "[2050]\tvalid_0's l1: 13.9098\tvalid_1's l1: 14.8197\n",
      "[2100]\tvalid_0's l1: 13.8891\tvalid_1's l1: 14.8169\n",
      "[2150]\tvalid_0's l1: 13.8675\tvalid_1's l1: 14.8141\n",
      "[2200]\tvalid_0's l1: 13.8471\tvalid_1's l1: 14.8126\n",
      "[2250]\tvalid_0's l1: 13.8272\tvalid_1's l1: 14.8114\n",
      "[2300]\tvalid_0's l1: 13.8062\tvalid_1's l1: 14.8095\n",
      "[2350]\tvalid_0's l1: 13.7864\tvalid_1's l1: 14.8086\n",
      "[2400]\tvalid_0's l1: 13.7663\tvalid_1's l1: 14.806\n",
      "[2450]\tvalid_0's l1: 13.7462\tvalid_1's l1: 14.8048\n",
      "[2500]\tvalid_0's l1: 13.7262\tvalid_1's l1: 14.8033\n",
      "[2550]\tvalid_0's l1: 13.7072\tvalid_1's l1: 14.8028\n",
      "[2600]\tvalid_0's l1: 13.6875\tvalid_1's l1: 14.8013\n",
      "[2650]\tvalid_0's l1: 13.6685\tvalid_1's l1: 14.8\n",
      "[2700]\tvalid_0's l1: 13.6496\tvalid_1's l1: 14.7996\n",
      "[2750]\tvalid_0's l1: 13.6313\tvalid_1's l1: 14.7982\n",
      "[2800]\tvalid_0's l1: 13.612\tvalid_1's l1: 14.7967\n",
      "[2850]\tvalid_0's l1: 13.5936\tvalid_1's l1: 14.7954\n",
      "[2900]\tvalid_0's l1: 13.5748\tvalid_1's l1: 14.7944\n",
      "[2950]\tvalid_0's l1: 13.5564\tvalid_1's l1: 14.7928\n",
      "[3000]\tvalid_0's l1: 13.5384\tvalid_1's l1: 14.7912\n",
      "[3050]\tvalid_0's l1: 13.5209\tvalid_1's l1: 14.7899\n",
      "[3100]\tvalid_0's l1: 13.5035\tvalid_1's l1: 14.7887\n",
      "[3150]\tvalid_0's l1: 13.4859\tvalid_1's l1: 14.7886\n",
      "[3200]\tvalid_0's l1: 13.4682\tvalid_1's l1: 14.7883\n",
      "[3250]\tvalid_0's l1: 13.4506\tvalid_1's l1: 14.7872\n",
      "[3300]\tvalid_0's l1: 13.4335\tvalid_1's l1: 14.7872\n",
      "[3350]\tvalid_0's l1: 13.4165\tvalid_1's l1: 14.7865\n",
      "[3400]\tvalid_0's l1: 13.3996\tvalid_1's l1: 14.7862\n",
      "[3450]\tvalid_0's l1: 13.3829\tvalid_1's l1: 14.7856\n",
      "[3500]\tvalid_0's l1: 13.3657\tvalid_1's l1: 14.7843\n",
      "[3550]\tvalid_0's l1: 13.3491\tvalid_1's l1: 14.7846\n",
      "[3600]\tvalid_0's l1: 13.3313\tvalid_1's l1: 14.7832\n",
      "[3650]\tvalid_0's l1: 13.3142\tvalid_1's l1: 14.7823\n",
      "[3700]\tvalid_0's l1: 13.2972\tvalid_1's l1: 14.7824\n",
      "[3750]\tvalid_0's l1: 13.2806\tvalid_1's l1: 14.7817\n",
      "[3800]\tvalid_0's l1: 13.2638\tvalid_1's l1: 14.7822\n",
      "[3850]\tvalid_0's l1: 13.2473\tvalid_1's l1: 14.7816\n",
      "[3900]\tvalid_0's l1: 13.2316\tvalid_1's l1: 14.7818\n",
      "[3950]\tvalid_0's l1: 13.2144\tvalid_1's l1: 14.781\n",
      "[4000]\tvalid_0's l1: 13.1988\tvalid_1's l1: 14.7809\n",
      "[4050]\tvalid_0's l1: 13.1827\tvalid_1's l1: 14.7808\n",
      "[4100]\tvalid_0's l1: 13.1666\tvalid_1's l1: 14.7809\n",
      "[4150]\tvalid_0's l1: 13.1507\tvalid_1's l1: 14.7815\n",
      "[4200]\tvalid_0's l1: 13.135\tvalid_1's l1: 14.7815\n",
      "Early stopping, best iteration is:\n",
      "[3964]\tvalid_0's l1: 13.21\tvalid_1's l1: 14.7804\n",
      "****************************************************************************************************\n",
      "MAE Model 0.06367346108002223\n",
      "MSE Model 0.06336988979602093\n",
      "Merge Model12 0.0637306555814696\n",
      "Training until validation scores don't improve for 250 rounds.\n",
      "[50]\tvalid_0's l1: 27.669\tvalid_1's l1: 27.877\n",
      "[100]\tvalid_0's l1: 23.9254\tvalid_1's l1: 24.1487\n",
      "[150]\tvalid_0's l1: 21.2515\tvalid_1's l1: 21.5161\n",
      "[200]\tvalid_0's l1: 19.3586\tvalid_1's l1: 19.6663\n",
      "[250]\tvalid_0's l1: 18.0328\tvalid_1's l1: 18.3855\n",
      "[300]\tvalid_0's l1: 17.118\tvalid_1's l1: 17.4855\n",
      "[350]\tvalid_0's l1: 16.4755\tvalid_1's l1: 16.8513\n",
      "[400]\tvalid_0's l1: 16.0147\tvalid_1's l1: 16.4019\n",
      "[450]\tvalid_0's l1: 15.6808\tvalid_1's l1: 16.091\n",
      "[500]\tvalid_0's l1: 15.4325\tvalid_1's l1: 15.8723\n",
      "[550]\tvalid_0's l1: 15.2422\tvalid_1's l1: 15.7119\n",
      "[600]\tvalid_0's l1: 15.093\tvalid_1's l1: 15.5912\n",
      "[650]\tvalid_0's l1: 14.9699\tvalid_1's l1: 15.4942\n",
      "[700]\tvalid_0's l1: 14.8639\tvalid_1's l1: 15.4103\n",
      "[750]\tvalid_0's l1: 14.7752\tvalid_1's l1: 15.347\n",
      "[800]\tvalid_0's l1: 14.6984\tvalid_1's l1: 15.2945\n",
      "[850]\tvalid_0's l1: 14.6314\tvalid_1's l1: 15.2494\n",
      "[900]\tvalid_0's l1: 14.5726\tvalid_1's l1: 15.211\n",
      "[950]\tvalid_0's l1: 14.518\tvalid_1's l1: 15.1773\n",
      "[1000]\tvalid_0's l1: 14.4682\tvalid_1's l1: 15.1481\n",
      "[1050]\tvalid_0's l1: 14.4224\tvalid_1's l1: 15.122\n",
      "[1100]\tvalid_0's l1: 14.3807\tvalid_1's l1: 15.1011\n",
      "[1150]\tvalid_0's l1: 14.3421\tvalid_1's l1: 15.0811\n",
      "[1200]\tvalid_0's l1: 14.3053\tvalid_1's l1: 15.0642\n",
      "[1250]\tvalid_0's l1: 14.2694\tvalid_1's l1: 15.0482\n",
      "[1300]\tvalid_0's l1: 14.2366\tvalid_1's l1: 15.0341\n",
      "[1350]\tvalid_0's l1: 14.206\tvalid_1's l1: 15.0232\n",
      "[1400]\tvalid_0's l1: 14.1753\tvalid_1's l1: 15.0119\n",
      "[1450]\tvalid_0's l1: 14.1464\tvalid_1's l1: 15.0022\n",
      "[1500]\tvalid_0's l1: 14.1184\tvalid_1's l1: 14.9927\n",
      "[1550]\tvalid_0's l1: 14.091\tvalid_1's l1: 14.985\n",
      "[1600]\tvalid_0's l1: 14.0657\tvalid_1's l1: 14.9782\n",
      "[1650]\tvalid_0's l1: 14.0414\tvalid_1's l1: 14.9727\n",
      "[1700]\tvalid_0's l1: 14.0171\tvalid_1's l1: 14.9666\n",
      "[1750]\tvalid_0's l1: 13.9927\tvalid_1's l1: 14.9609\n",
      "[1800]\tvalid_0's l1: 13.9691\tvalid_1's l1: 14.9562\n",
      "[1850]\tvalid_0's l1: 13.9445\tvalid_1's l1: 14.9499\n",
      "[1900]\tvalid_0's l1: 13.9207\tvalid_1's l1: 14.9454\n",
      "[1950]\tvalid_0's l1: 13.8992\tvalid_1's l1: 14.9415\n",
      "[2000]\tvalid_0's l1: 13.8783\tvalid_1's l1: 14.9375\n",
      "[2050]\tvalid_0's l1: 13.8578\tvalid_1's l1: 14.9345\n",
      "[2100]\tvalid_0's l1: 13.8373\tvalid_1's l1: 14.9314\n",
      "[2150]\tvalid_0's l1: 13.8174\tvalid_1's l1: 14.9289\n",
      "[2200]\tvalid_0's l1: 13.7977\tvalid_1's l1: 14.9256\n",
      "[2250]\tvalid_0's l1: 13.7782\tvalid_1's l1: 14.9227\n",
      "[2300]\tvalid_0's l1: 13.7574\tvalid_1's l1: 14.9186\n",
      "[2350]\tvalid_0's l1: 13.7395\tvalid_1's l1: 14.9168\n",
      "[2400]\tvalid_0's l1: 13.7196\tvalid_1's l1: 14.9136\n",
      "[2450]\tvalid_0's l1: 13.7013\tvalid_1's l1: 14.9112\n",
      "[2500]\tvalid_0's l1: 13.6819\tvalid_1's l1: 14.9077\n",
      "[2550]\tvalid_0's l1: 13.6632\tvalid_1's l1: 14.9048\n",
      "[2600]\tvalid_0's l1: 13.6444\tvalid_1's l1: 14.9033\n",
      "[2650]\tvalid_0's l1: 13.6256\tvalid_1's l1: 14.9012\n",
      "[2700]\tvalid_0's l1: 13.6069\tvalid_1's l1: 14.8982\n",
      "[2750]\tvalid_0's l1: 13.5899\tvalid_1's l1: 14.8965\n",
      "[2800]\tvalid_0's l1: 13.5731\tvalid_1's l1: 14.8946\n",
      "[2850]\tvalid_0's l1: 13.5561\tvalid_1's l1: 14.8922\n",
      "[2900]\tvalid_0's l1: 13.5378\tvalid_1's l1: 14.8905\n",
      "[2950]\tvalid_0's l1: 13.5204\tvalid_1's l1: 14.8886\n",
      "[3000]\tvalid_0's l1: 13.5034\tvalid_1's l1: 14.8868\n",
      "[3050]\tvalid_0's l1: 13.4859\tvalid_1's l1: 14.8841\n",
      "[3100]\tvalid_0's l1: 13.4693\tvalid_1's l1: 14.8823\n",
      "[3150]\tvalid_0's l1: 13.4514\tvalid_1's l1: 14.8811\n",
      "[3200]\tvalid_0's l1: 13.435\tvalid_1's l1: 14.8801\n",
      "[3250]\tvalid_0's l1: 13.4183\tvalid_1's l1: 14.878\n",
      "[3300]\tvalid_0's l1: 13.4016\tvalid_1's l1: 14.8772\n",
      "[3350]\tvalid_0's l1: 13.3863\tvalid_1's l1: 14.8758\n",
      "[3400]\tvalid_0's l1: 13.3705\tvalid_1's l1: 14.8745\n",
      "[3450]\tvalid_0's l1: 13.3552\tvalid_1's l1: 14.8739\n",
      "[3500]\tvalid_0's l1: 13.3388\tvalid_1's l1: 14.8737\n",
      "[3550]\tvalid_0's l1: 13.3239\tvalid_1's l1: 14.8729\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3600]\tvalid_0's l1: 13.3081\tvalid_1's l1: 14.8713\n",
      "[3650]\tvalid_0's l1: 13.2928\tvalid_1's l1: 14.8702\n",
      "[3700]\tvalid_0's l1: 13.2764\tvalid_1's l1: 14.8686\n",
      "[3750]\tvalid_0's l1: 13.2611\tvalid_1's l1: 14.8675\n",
      "[3800]\tvalid_0's l1: 13.2459\tvalid_1's l1: 14.8664\n",
      "[3850]\tvalid_0's l1: 13.2323\tvalid_1's l1: 14.8663\n",
      "[3900]\tvalid_0's l1: 13.2177\tvalid_1's l1: 14.8652\n",
      "[3950]\tvalid_0's l1: 13.2031\tvalid_1's l1: 14.864\n",
      "[4000]\tvalid_0's l1: 13.1898\tvalid_1's l1: 14.8636\n",
      "[4050]\tvalid_0's l1: 13.176\tvalid_1's l1: 14.8621\n",
      "[4100]\tvalid_0's l1: 13.161\tvalid_1's l1: 14.8613\n",
      "[4150]\tvalid_0's l1: 13.1467\tvalid_1's l1: 14.8602\n",
      "[4200]\tvalid_0's l1: 13.1336\tvalid_1's l1: 14.8596\n",
      "[4250]\tvalid_0's l1: 13.121\tvalid_1's l1: 14.8586\n",
      "[4300]\tvalid_0's l1: 13.1086\tvalid_1's l1: 14.8581\n",
      "[4350]\tvalid_0's l1: 13.0945\tvalid_1's l1: 14.8569\n",
      "[4400]\tvalid_0's l1: 13.081\tvalid_1's l1: 14.8556\n",
      "[4450]\tvalid_0's l1: 13.0689\tvalid_1's l1: 14.8553\n",
      "[4500]\tvalid_0's l1: 13.0565\tvalid_1's l1: 14.8553\n",
      "[4550]\tvalid_0's l1: 13.045\tvalid_1's l1: 14.8546\n",
      "[4600]\tvalid_0's l1: 13.0338\tvalid_1's l1: 14.854\n",
      "[4650]\tvalid_0's l1: 13.0214\tvalid_1's l1: 14.8534\n",
      "[4700]\tvalid_0's l1: 13.0083\tvalid_1's l1: 14.8528\n",
      "[4750]\tvalid_0's l1: 12.9964\tvalid_1's l1: 14.8525\n",
      "[4800]\tvalid_0's l1: 12.983\tvalid_1's l1: 14.8513\n",
      "[4850]\tvalid_0's l1: 12.9715\tvalid_1's l1: 14.8512\n",
      "[4900]\tvalid_0's l1: 12.9584\tvalid_1's l1: 14.8507\n",
      "[4950]\tvalid_0's l1: 12.9463\tvalid_1's l1: 14.8504\n",
      "[5000]\tvalid_0's l1: 12.9343\tvalid_1's l1: 14.8497\n",
      "[5050]\tvalid_0's l1: 12.9215\tvalid_1's l1: 14.849\n",
      "[5100]\tvalid_0's l1: 12.909\tvalid_1's l1: 14.8483\n",
      "[5150]\tvalid_0's l1: 12.8961\tvalid_1's l1: 14.8477\n",
      "[5200]\tvalid_0's l1: 12.8844\tvalid_1's l1: 14.8475\n",
      "[5250]\tvalid_0's l1: 12.8721\tvalid_1's l1: 14.8471\n",
      "[5300]\tvalid_0's l1: 12.8612\tvalid_1's l1: 14.8469\n",
      "[5350]\tvalid_0's l1: 12.8525\tvalid_1's l1: 14.8466\n",
      "[5400]\tvalid_0's l1: 12.8417\tvalid_1's l1: 14.8464\n",
      "[5450]\tvalid_0's l1: 12.8295\tvalid_1's l1: 14.8458\n",
      "[5500]\tvalid_0's l1: 12.8178\tvalid_1's l1: 14.8447\n",
      "[5550]\tvalid_0's l1: 12.8071\tvalid_1's l1: 14.8441\n",
      "[5600]\tvalid_0's l1: 12.7954\tvalid_1's l1: 14.8436\n",
      "[5650]\tvalid_0's l1: 12.7833\tvalid_1's l1: 14.8435\n",
      "[5700]\tvalid_0's l1: 12.7728\tvalid_1's l1: 14.843\n",
      "[5750]\tvalid_0's l1: 12.7627\tvalid_1's l1: 14.8429\n",
      "[5800]\tvalid_0's l1: 12.751\tvalid_1's l1: 14.8422\n",
      "[5850]\tvalid_0's l1: 12.7398\tvalid_1's l1: 14.8414\n",
      "[5900]\tvalid_0's l1: 12.7286\tvalid_1's l1: 14.8411\n",
      "[5950]\tvalid_0's l1: 12.7176\tvalid_1's l1: 14.8404\n",
      "[6000]\tvalid_0's l1: 12.7078\tvalid_1's l1: 14.8406\n",
      "[6050]\tvalid_0's l1: 12.6956\tvalid_1's l1: 14.8403\n",
      "[6100]\tvalid_0's l1: 12.6841\tvalid_1's l1: 14.8396\n",
      "[6150]\tvalid_0's l1: 12.6741\tvalid_1's l1: 14.8398\n",
      "[6200]\tvalid_0's l1: 12.6628\tvalid_1's l1: 14.8395\n",
      "[6250]\tvalid_0's l1: 12.6512\tvalid_1's l1: 14.8392\n",
      "[6300]\tvalid_0's l1: 12.6407\tvalid_1's l1: 14.839\n",
      "[6350]\tvalid_0's l1: 12.6292\tvalid_1's l1: 14.8391\n",
      "[6400]\tvalid_0's l1: 12.6179\tvalid_1's l1: 14.8387\n",
      "[6450]\tvalid_0's l1: 12.6075\tvalid_1's l1: 14.8384\n",
      "[6500]\tvalid_0's l1: 12.5972\tvalid_1's l1: 14.838\n",
      "[6550]\tvalid_0's l1: 12.588\tvalid_1's l1: 14.8377\n",
      "[6600]\tvalid_0's l1: 12.5782\tvalid_1's l1: 14.8376\n",
      "[6650]\tvalid_0's l1: 12.5688\tvalid_1's l1: 14.8377\n",
      "[6700]\tvalid_0's l1: 12.5587\tvalid_1's l1: 14.8371\n",
      "[6750]\tvalid_0's l1: 12.5488\tvalid_1's l1: 14.8369\n",
      "[6800]\tvalid_0's l1: 12.5387\tvalid_1's l1: 14.8369\n",
      "[6850]\tvalid_0's l1: 12.5303\tvalid_1's l1: 14.8367\n",
      "[6900]\tvalid_0's l1: 12.5213\tvalid_1's l1: 14.8367\n",
      "[6950]\tvalid_0's l1: 12.5138\tvalid_1's l1: 14.8366\n",
      "[7000]\tvalid_0's l1: 12.5076\tvalid_1's l1: 14.8365\n",
      "[7050]\tvalid_0's l1: 12.5\tvalid_1's l1: 14.8362\n",
      "[7100]\tvalid_0's l1: 12.4912\tvalid_1's l1: 14.8364\n",
      "[7150]\tvalid_0's l1: 12.4821\tvalid_1's l1: 14.8361\n",
      "[7200]\tvalid_0's l1: 12.4736\tvalid_1's l1: 14.8356\n",
      "[7250]\tvalid_0's l1: 12.4675\tvalid_1's l1: 14.8356\n",
      "[7300]\tvalid_0's l1: 12.4616\tvalid_1's l1: 14.8359\n",
      "[7350]\tvalid_0's l1: 12.4552\tvalid_1's l1: 14.8362\n",
      "[7400]\tvalid_0's l1: 12.4492\tvalid_1's l1: 14.8361\n",
      "[7450]\tvalid_0's l1: 12.4433\tvalid_1's l1: 14.8359\n",
      "Early stopping, best iteration is:\n",
      "[7215]\tvalid_0's l1: 12.4721\tvalid_1's l1: 14.8354\n",
      "Training until validation scores don't improve for 250 rounds.\n",
      "[50]\tvalid_0's l1: 28.1164\tvalid_1's l1: 28.2198\n",
      "[100]\tvalid_0's l1: 24.13\tvalid_1's l1: 24.2218\n",
      "[150]\tvalid_0's l1: 21.2933\tvalid_1's l1: 21.4102\n",
      "[200]\tvalid_0's l1: 19.317\tvalid_1's l1: 19.4706\n",
      "[250]\tvalid_0's l1: 17.9627\tvalid_1's l1: 18.1459\n",
      "[300]\tvalid_0's l1: 17.0459\tvalid_1's l1: 17.2604\n",
      "[350]\tvalid_0's l1: 16.4151\tvalid_1's l1: 16.6631\n",
      "[400]\tvalid_0's l1: 15.9734\tvalid_1's l1: 16.2575\n",
      "[450]\tvalid_0's l1: 15.6593\tvalid_1's l1: 15.9773\n",
      "[500]\tvalid_0's l1: 15.4287\tvalid_1's l1: 15.7735\n",
      "[550]\tvalid_0's l1: 15.2529\tvalid_1's l1: 15.624\n",
      "[600]\tvalid_0's l1: 15.1166\tvalid_1's l1: 15.512\n",
      "[650]\tvalid_0's l1: 15.0064\tvalid_1's l1: 15.4246\n",
      "[700]\tvalid_0's l1: 14.9107\tvalid_1's l1: 15.3505\n",
      "[750]\tvalid_0's l1: 14.8247\tvalid_1's l1: 15.2885\n",
      "[800]\tvalid_0's l1: 14.7511\tvalid_1's l1: 15.2363\n",
      "[850]\tvalid_0's l1: 14.686\tvalid_1's l1: 15.1922\n",
      "[900]\tvalid_0's l1: 14.6277\tvalid_1's l1: 15.154\n",
      "[950]\tvalid_0's l1: 14.5752\tvalid_1's l1: 15.1209\n",
      "[1000]\tvalid_0's l1: 14.5266\tvalid_1's l1: 15.092\n",
      "[1050]\tvalid_0's l1: 14.4828\tvalid_1's l1: 15.0659\n",
      "[1100]\tvalid_0's l1: 14.4402\tvalid_1's l1: 15.0427\n",
      "[1150]\tvalid_0's l1: 14.4009\tvalid_1's l1: 15.0216\n",
      "[1200]\tvalid_0's l1: 14.3635\tvalid_1's l1: 15.0037\n",
      "[1250]\tvalid_0's l1: 14.3271\tvalid_1's l1: 14.9869\n",
      "[1300]\tvalid_0's l1: 14.2932\tvalid_1's l1: 14.973\n",
      "[1350]\tvalid_0's l1: 14.2621\tvalid_1's l1: 14.9599\n",
      "[1400]\tvalid_0's l1: 14.2322\tvalid_1's l1: 14.9474\n",
      "[1450]\tvalid_0's l1: 14.2029\tvalid_1's l1: 14.9366\n",
      "[1500]\tvalid_0's l1: 14.1748\tvalid_1's l1: 14.9268\n",
      "[1550]\tvalid_0's l1: 14.1485\tvalid_1's l1: 14.9191\n",
      "[1600]\tvalid_0's l1: 14.1229\tvalid_1's l1: 14.9102\n",
      "[1650]\tvalid_0's l1: 14.0984\tvalid_1's l1: 14.9031\n",
      "[1700]\tvalid_0's l1: 14.0743\tvalid_1's l1: 14.8962\n",
      "[1750]\tvalid_0's l1: 14.0506\tvalid_1's l1: 14.8904\n",
      "[1800]\tvalid_0's l1: 14.0271\tvalid_1's l1: 14.8846\n",
      "[1850]\tvalid_0's l1: 14.0047\tvalid_1's l1: 14.879\n",
      "[1900]\tvalid_0's l1: 13.9822\tvalid_1's l1: 14.8743\n",
      "[1950]\tvalid_0's l1: 13.9605\tvalid_1's l1: 14.8701\n",
      "[2000]\tvalid_0's l1: 13.9398\tvalid_1's l1: 14.8661\n",
      "[2050]\tvalid_0's l1: 13.9187\tvalid_1's l1: 14.8626\n",
      "[2100]\tvalid_0's l1: 13.8988\tvalid_1's l1: 14.8599\n",
      "[2150]\tvalid_0's l1: 13.8781\tvalid_1's l1: 14.855\n",
      "[2200]\tvalid_0's l1: 13.8574\tvalid_1's l1: 14.8508\n",
      "[2250]\tvalid_0's l1: 13.8373\tvalid_1's l1: 14.8477\n",
      "[2300]\tvalid_0's l1: 13.8171\tvalid_1's l1: 14.8455\n",
      "[2350]\tvalid_0's l1: 13.7969\tvalid_1's l1: 14.8428\n",
      "[2400]\tvalid_0's l1: 13.777\tvalid_1's l1: 14.8406\n",
      "[2450]\tvalid_0's l1: 13.7574\tvalid_1's l1: 14.8375\n",
      "[2500]\tvalid_0's l1: 13.7381\tvalid_1's l1: 14.8341\n",
      "[2550]\tvalid_0's l1: 13.7189\tvalid_1's l1: 14.8323\n",
      "[2600]\tvalid_0's l1: 13.6999\tvalid_1's l1: 14.8299\n",
      "[2650]\tvalid_0's l1: 13.6814\tvalid_1's l1: 14.8273\n",
      "[2700]\tvalid_0's l1: 13.6632\tvalid_1's l1: 14.8252\n",
      "[2750]\tvalid_0's l1: 13.6444\tvalid_1's l1: 14.8236\n",
      "[2800]\tvalid_0's l1: 13.626\tvalid_1's l1: 14.8216\n",
      "[2850]\tvalid_0's l1: 13.6082\tvalid_1's l1: 14.8205\n",
      "[2900]\tvalid_0's l1: 13.5896\tvalid_1's l1: 14.8181\n",
      "[2950]\tvalid_0's l1: 13.572\tvalid_1's l1: 14.8163\n",
      "[3000]\tvalid_0's l1: 13.554\tvalid_1's l1: 14.8143\n",
      "[3050]\tvalid_0's l1: 13.5355\tvalid_1's l1: 14.8125\n",
      "[3100]\tvalid_0's l1: 13.5182\tvalid_1's l1: 14.8115\n",
      "[3150]\tvalid_0's l1: 13.5006\tvalid_1's l1: 14.8096\n",
      "[3200]\tvalid_0's l1: 13.483\tvalid_1's l1: 14.8092\n",
      "[3250]\tvalid_0's l1: 13.465\tvalid_1's l1: 14.8074\n",
      "[3300]\tvalid_0's l1: 13.4476\tvalid_1's l1: 14.8057\n",
      "[3350]\tvalid_0's l1: 13.4304\tvalid_1's l1: 14.8041\n",
      "[3400]\tvalid_0's l1: 13.4143\tvalid_1's l1: 14.8037\n",
      "[3450]\tvalid_0's l1: 13.3975\tvalid_1's l1: 14.8029\n",
      "[3500]\tvalid_0's l1: 13.381\tvalid_1's l1: 14.8016\n",
      "[3550]\tvalid_0's l1: 13.3639\tvalid_1's l1: 14.8011\n",
      "[3600]\tvalid_0's l1: 13.3472\tvalid_1's l1: 14.8001\n",
      "[3650]\tvalid_0's l1: 13.3308\tvalid_1's l1: 14.7991\n",
      "[3700]\tvalid_0's l1: 13.314\tvalid_1's l1: 14.7975\n",
      "[3750]\tvalid_0's l1: 13.297\tvalid_1's l1: 14.7967\n",
      "[3800]\tvalid_0's l1: 13.2807\tvalid_1's l1: 14.7961\n",
      "[3850]\tvalid_0's l1: 13.2641\tvalid_1's l1: 14.7948\n",
      "[3900]\tvalid_0's l1: 13.2482\tvalid_1's l1: 14.7945\n",
      "[3950]\tvalid_0's l1: 13.2319\tvalid_1's l1: 14.7933\n",
      "[4000]\tvalid_0's l1: 13.2164\tvalid_1's l1: 14.7928\n",
      "[4050]\tvalid_0's l1: 13.1998\tvalid_1's l1: 14.7926\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4100]\tvalid_0's l1: 13.1836\tvalid_1's l1: 14.7917\n",
      "[4150]\tvalid_0's l1: 13.1677\tvalid_1's l1: 14.7913\n",
      "[4200]\tvalid_0's l1: 13.1518\tvalid_1's l1: 14.7915\n",
      "[4250]\tvalid_0's l1: 13.1356\tvalid_1's l1: 14.7907\n",
      "[4300]\tvalid_0's l1: 13.1195\tvalid_1's l1: 14.7898\n",
      "[4350]\tvalid_0's l1: 13.1041\tvalid_1's l1: 14.7893\n",
      "[4400]\tvalid_0's l1: 13.0882\tvalid_1's l1: 14.7889\n",
      "[4450]\tvalid_0's l1: 13.0727\tvalid_1's l1: 14.7886\n",
      "[4500]\tvalid_0's l1: 13.058\tvalid_1's l1: 14.7886\n",
      "[4550]\tvalid_0's l1: 13.0423\tvalid_1's l1: 14.7882\n",
      "[4600]\tvalid_0's l1: 13.0271\tvalid_1's l1: 14.7879\n",
      "[4650]\tvalid_0's l1: 13.0118\tvalid_1's l1: 14.7879\n",
      "[4700]\tvalid_0's l1: 12.9967\tvalid_1's l1: 14.7874\n",
      "[4750]\tvalid_0's l1: 12.9819\tvalid_1's l1: 14.7866\n",
      "[4800]\tvalid_0's l1: 12.9668\tvalid_1's l1: 14.7863\n",
      "[4850]\tvalid_0's l1: 12.9519\tvalid_1's l1: 14.7856\n",
      "[4900]\tvalid_0's l1: 12.937\tvalid_1's l1: 14.7853\n",
      "[4950]\tvalid_0's l1: 12.9221\tvalid_1's l1: 14.786\n",
      "[5000]\tvalid_0's l1: 12.9067\tvalid_1's l1: 14.7854\n",
      "[5050]\tvalid_0's l1: 12.8918\tvalid_1's l1: 14.7851\n",
      "[5100]\tvalid_0's l1: 12.8767\tvalid_1's l1: 14.7839\n",
      "[5150]\tvalid_0's l1: 12.8615\tvalid_1's l1: 14.7839\n",
      "[5200]\tvalid_0's l1: 12.8468\tvalid_1's l1: 14.7834\n",
      "[5250]\tvalid_0's l1: 12.8326\tvalid_1's l1: 14.7834\n",
      "[5300]\tvalid_0's l1: 12.8178\tvalid_1's l1: 14.7829\n",
      "[5350]\tvalid_0's l1: 12.8032\tvalid_1's l1: 14.7832\n",
      "[5400]\tvalid_0's l1: 12.7892\tvalid_1's l1: 14.7833\n",
      "[5450]\tvalid_0's l1: 12.7744\tvalid_1's l1: 14.7839\n",
      "[5500]\tvalid_0's l1: 12.7595\tvalid_1's l1: 14.7833\n",
      "[5550]\tvalid_0's l1: 12.7445\tvalid_1's l1: 14.7831\n",
      "Early stopping, best iteration is:\n",
      "[5311]\tvalid_0's l1: 12.8146\tvalid_1's l1: 14.7827\n",
      "****************************************************************************************************\n",
      "MAE Model 0.06314959550420497\n",
      "MSE Model 0.06336060282511229\n",
      "Merge Model12 0.06332385690501188\n"
     ]
    }
   ],
   "source": [
    "models_mae, models_mse, importances   = _get_values_lgbregresser_models(train_fea[fea_cols].values, train_fea['信用分'].values, feature_names=fea_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型验证结果\n",
    "### 直接使用MAE指标的线下5-fold验证结果(0.06349584)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.06349584922088962"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean([0.06287124227830537,0.06356345258514168,0.06422149465677383, 0.06367346108002223,0.06314959550420497])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用MAE+MSE结合的线下5-fold验证结果(0.0636108)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.06361087588322085"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean([0.063039062547442,0.06359245145019841,0.06436835293198238, 0.0637306555814696, 0.06332385690501188])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 模型测试&提交\n",
    "## MAE提交（0.06354）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_mae = 0\n",
    "for i,model in enumerate(models_mae): \n",
    "    pred_mae += model.predict(test_fea[fea_cols]) * 0.2\n",
    "test_fea['pred_mae'] = pred_mae"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pred_mse = 0\n",
    "for i,model in enumerate(models_mse): \n",
    "    pred_mse += model.predict(test_fea[fea_cols]) * 0.2\n",
    "test_fea['pred_mse'] = pred_mse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "submit_mae = pd.DataFrame()\n",
    "submit_mae['id']    = test_fea['用户编码'].values\n",
    "submit_mae['score'] = test_fea['pred_mae'].values \n",
    "submit_mae['score'] = submit_mae['score'].astype(int)\n",
    "submit_mae[['id','score']].to_csv('baseline_mae.csv',index = None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    50000.000000\n",
       "mean       618.780040\n",
       "std         37.730256\n",
       "min        476.000000\n",
       "25%        598.000000\n",
       "50%        628.000000\n",
       "75%        646.000000\n",
       "max        696.000000\n",
       "Name: score, dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submit_mae['score'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## MSE提交（0.06359）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "test_fea = test_fea.sort_values('pred_mae')\n",
    "test_fea['ranks'] = list(range(test_fea.shape[0]))\n",
    "test_fea['score'] = test_fea['pred_mae'].values\n",
    "test_fea.loc[test_fea.ranks<10000,'score']  = test_fea.loc[test_fea.ranks< 10000,'pred_mse'].values *0.4 + test_fea.loc[test_fea.ranks< 10000,'pred_mae'].values * 0.6\n",
    "test_fea.loc[test_fea.ranks>40000,'score']  = test_fea.loc[test_fea.ranks> 40000,'pred_mse'].values *0.4 + test_fea.loc[test_fea.ranks> 40000,'pred_mae'].values * 0.6\n",
    "         "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "submit_mae_mse = pd.DataFrame()\n",
    "submit_mae_mse['id']    = test_fea['用户编码'].values\n",
    "submit_mae_mse['score'] = test_fea['score'].values \n",
    "submit_mae_mse['score'] = submit_mae_mse['score'].astype(int)\n",
    "submit_mae_mse[['id','score']].to_csv('baseline_mae_mse.csv',index = None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    50000.000000\n",
       "mean       618.586220\n",
       "std         37.896524\n",
       "min        471.000000\n",
       "25%        598.000000\n",
       "50%        628.000000\n",
       "75%        646.000000\n",
       "max        695.000000\n",
       "Name: score, dtype: float64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submit_mae_mse['score'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 结论\n",
    "\n",
    "结论不多说,我们看下面的分数就行,和我们线下验证的结果是一致的,如果这边文章有帮助到您,欢迎<font color=red>转发or喜欢作者</font>.....\n",
    "\n",
    "![](./pic/compare.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
